SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[37] Ingmar Skoog and Deborah Gustafson. Update on hypertension and alzheimer's disease. Neurological Research, 28(6):605-611, 2006.
[38] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
[39] Jin Tian and Judea Pearl. A general identification condition for causal effects. eScholarship, University of California, 2002.
[40] Zhi Wang, Chicheng Zhang, and Kamalika Chaudhuri. Thompson sampling for robust transfer in multi-task bandits. arXiv preprint arXiv:2206.08556, 2022.
[41] Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In International Conference on Artificial Intelligence and Statistics, pages 1531-1539. PMLR, 2021.
[42] Kan Xu and Hamsa Bastani. Learning across bandits in high dimension via robust statistics. arXiv preprint arXiv:2112.14233, 2021.
[43] Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandit: a causal approach. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 1778–1780, 2017.
[44] Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. arXiv preprint arXiv:2110.05690, 2021.
Appendix for "Transportability for Bandits with Data from Different Environments"
This Appendix includes
- Related work in Appendix A.
- Discussion on limitations and analysis of misspecification in Appendix B.
- Proofs in Appendix C.
- Details on the data generating mechanisms Appendix D.
- Details on the Gibbs sampler in Appendix E.
In the bandit literature, problems of transfer learning in which existing experience and knowledge is used to improve the performance of an agent appear under the rubric of multi-task learning and meta-learning. Typically, in the context of multi-task learning the agent aims to solve a prescribed set of related bandit tasks with shared structure. For example, [41] consider the setting in which multiple players interact in an environment with the property that each player has slightly different associated reward distributions. [40] extend this approach to the Thompson sampling algorithm with an assumption that reward distribution between players are close but not identical. Similar ideas can be found in the contextual case in which, e.g. [13] propose to leverage similarities in contexts for different arms and improve prediction of reward distributions from contexts. In the context of meta-learning, the agent is designed to work well on an arbitrary number of tasks from a common environment (i.e. sampled from a prescribed distribution), relying on already completed tasks from the same environment. For example, [4, 20, 33] assume a hierarchical bandit structure in which parameters governing multiple bandit instances are sampled i.i.d. from a meta-prior. The authors demonstrate that updates on the meta-prior from one instance can benefit other instances and that regret bounds over a sequence of instances can be established. [30] adopt a similar setting to infer a Lipschitz continuity constant which can be used to derive scale free regret bounds. Others, e.g. [10, 42], have assumed that structural parameters of each instance can be decomposed into a shared component and an instance-specific component, and learned with high-dimensional regularization schemes.
Existing methods adopt a relatively wide range of assumptions on the "relatedness" of different bandit instances or players that include, as surveyed above, the existence of a meta-distribution across bandit instances, assumptions on the pairwise similarities in the reward distributions of different players, assumptions on the decomposition of bandit parameters, and assumptions on the distribution of contextual variables and their association with rewards. The environment and causal structure typically remain invariant across bandit instances, and only small and specific set of changes in distribution are allowed across bandit instances.
The proposed approach aims to tackle a more general setting in which we allow for arbitrary changes in the causal mechanisms underlying each environment as long as their location can be established and encoded in selection diagrams, i.e. graphs that describe the difference in structure across domains without constraining their form. In the causality literature, these analyses fall under the transportability umbrella. Several authors have demonstrated the power of this approach to identify causal effects, predict counterfactual distributions and, more generally, make inference across different environments [3, 11, 32, 5]. In the bandit literature, there is also an extensive literature on various ways of exploiting knowledge of a causal graph for a single environment, known as causal bandits [23, 36, 28, 27, 24, 26, 12, 29, 7, 18]. Within this literature the causal graph specifies a dependency structure between different actions that allows an agent to identify redundant actions and ultimately more efficient exploration, improving regret bounds by a multiplicative factor. Practical examples of the use of causal graphs (within decision-making problems) are prevalent in the contexts of clinical trials, healthcare, and advertising. We note, in particular, the examples in Figs. 2, 3, 4, in [22] that encode the design of case-cohort studies and clinical trials used in the MORGAM study [14] using causal graphs for the estimation of causal effects. Separately, [21] consider the
International Stroke Trial and the known causal associations between relevant features for policy optimization. In the context of advertising, [8] describes the use of causal methods with several detailed examples. Specifically, Figs. 3, 4, and 6 in [8] show examples of causal graphs that may be defined for particular computational advertising applications. See also [5] for a bandit algorithm leveraging the computational advertising graphs in [8], and [36, Sec. 5.2] for similar causal treatments also in the context of advertising. So far, however, all bandit algorithms exploiting causal knowledge have been studied in the "single environment" setting without access to prior data. Variations among existing proposals are due to the class of graphs or the extent of knowledge of the graph, e.g. with or without knowledge of the full causal graph, with or without unobserved confounding, etc. One may interpret the proposed approach as extending the causal bandit formalism to problems in which data from multiple different environments is available prior to an agent experimenting in a target environment, all of which are, as in the causal bandit literature, described by underlying causal models.
(a)  $P^{*}(z)$  arbitrary.
(b)  $P^{*}(z)\in [0.2,0.8]$
(c)  $P^{*}(z)\in [0.3,0.7]$
Figure 6: Prior expected reward distributions under misspecification of  $Z$ .
(d)  $P^{*}(z)\in [0.4,0.6]$
B Limitations and further analysis
Knowledge of structure may detract from the appeal of active experimentation and might limit the extent to which the proposed approach can be applied. Although in practice several techniques can be used to alleviate this problem such as causal discovery or considering multiple causal graphs as potential causal explanations for the phenomenon of interest, significant domain knowledge will in general be necessary for consistently using prior data. This observation holds for causal bandits more generally.
B.1 Misspecification
Some degree of misspecification, however, can be handled without voiding guarantees on performance improvements, especially when the assumed causal graphs and selection diagrams form super-models for the underlying phenomenon of interest.
Starting with causal graphs: a causal graph  $\tilde{\mathcal{G}}^{a,b}$  defined as a graph  $(\mathbf{V},\tilde{\mathcal{E}})$  is said to be a super-model of a causal graph  $\mathcal{G} = (\mathbf{V},\mathcal{E})$ $\mathcal{E}\subseteq \tilde{\mathcal{E}}$  . For any causal graph  $\mathcal{G}$  it holds that adding an edge, creating a new graph  $\tilde{\mathcal{G}}$  , creates a larger set of SCMs  $\mathcal{M}(\tilde{\mathcal{G}})$  that subsumes  $\mathcal{M}(\mathcal{G})$  , i.e.  $\mathcal{M}(\tilde{\mathcal{G}})\subset \mathcal{M}(\tilde{\mathcal{G}})$  , and as a consequence any SCM with non-zero mass under  $P_{\mathcal{M}(\mathcal{G})}$  also has non-zero mass under  $P_{\mathcal{M}(\tilde{\mathcal{G}})}$  . This implies that prior distributions  $P_{M\sim \mathcal{M}(\tilde{\mathcal{G}})}(\mathbb{E}_{P_M}[Y_x]\mid \bar{\boldsymbol{v}})$  remain valid whenever the causal graph  $\tilde{\mathcal{G}}$  is a super-model of the true underlying graph  $\mathcal{G}$  as the space of models defined by  $\mathcal{G}$  is included in the space of models defined  $\tilde{\mathcal{G}}$  . For example, consider a scenario in which source and target domains are equal with an underlying causal graph given by  $\mathcal{G}\coloneqq \{X\to Y\}$  . Prior to experimentation with knowledge of the graph and source data distribution  $P(x,y)$  , reward probabilities under agent intervention can be uniquely computed, i.e.  $P(y\mid do(x)) = P(y\mid x)$  . However, if we are unsure about the presence of an unobserved confounder between  $X$  and  $Y$  , we may instead consider a supermodel  $\mathcal{G}\coloneqq \{X\rightarrow Y,X\leftarrow \dots \rightarrow Y\}$  under which reward probabilities under agent intervention can be determined to be distributed in the interval  $[P(x,y),P(x,y) + 1 - P(x)]$  , that, in particular, includes the true value  $P(y\mid do(x)) = P(y\mid x)$  . Inference based on  $\tilde{\mathcal{G}}$  instead of  $\mathcal{G}$  leads to less informative reward probabilities but remains correct and informative with respect to methods not leveraging prior data as  $0\leqslant P(x,y)\leqslant P(y\mid do(x))\leqslant P(x,y) + 1 - P(x)\leqslant 1$
A similar reasoning can be applied to selection diagrams and selection nodes, as the latter only indicates a potential discrepancy across domains. A selection diagram  $\tilde{\mathcal{G}}^{a,b}$  defined as a graph  $(\mathbf{V} \cup \tilde{\mathbf{S}}, \tilde{\mathcal{E}})$  is said to be a super-model of a selection diagram  $\mathcal{G}^{a,b} = (\mathbf{V} \cup \mathbf{S}, \mathcal{E})$  if  $\mathbf{S} \subseteq \tilde{\mathbf{S}}$  and  $\mathcal{E} \subseteq \tilde{\mathcal{E}}$ . In words, super-models can have more edges or selection nodes. If a researcher is unsure of whether a causal mechanism differs or not, they may still conduct correct
inference by being conservative and assuming that a discrepancy exists. Under these types of misspecification, where assumed selection diagrams represent super-models of the underlying data generating process all claims and propositions hold and we can guarantee to never under-perform methods that do not consider prior data or their structure. For example, consider a system with binary variables  $X,Y,Z\in \{0,1\}$  whose causal association and differences across domains are given by Fig. 7. In reality, target and source environments coincide and, with access to batch data from the source environment  $\pi^a$ , one could uniquely identify expected rewards by computing  $\mathbb{E}_{P*}Y_{x} = \sum_{y}y\sum_{z}P^{a}(y\mid x,z)P^{a}(z)$ . However, we suspect some degree of discrepancy in the
Figure 7:  $\mathcal{G}^{a,*}$
(c)
Figure 9: Run time experiments.
causal assignment of  $Z$  across environments, and will consider inference of expected rewards under various degrees of discrepancy, e.g. one may explicitly assume that either  $P^{*}(z)$  is arbitrary or for example assume  $P^{*}(z) \in [a,b]$ . We illustrate various such scenarios in Fig. 6. Each panel, from right to left increases allows for a greater level of uncertainty that results in a correspondingly wider distribution of expected reward prior to experimentation. The corresponding regret of bandit algorithms that incorporate these bounds on  $Z$  in  $P^{*}$  to warm start the distribution of expected regret prior to experimentation are given in Fig. 8. We observe that the wider the prior the higher the corresponding regret, which illustrates how degrees of prior knowledge influence regret.
Using the same reasoning, it holds that if instead the researcher misses edges or incorrectly assigns priors on the causal mechanisms to be expected in the target domain, prior inference will be incorrect in general. For example, if distributions of expected reward in Fig. 6 do not cover the true expected reward, then a bandit algorithm will need to correct for such a biased prior and we can expect cumulative rewards to be lower than those of an algorithm that ignores prior data and structure. The extent to which the corresponding regret of a bandit algorithm incorrectly making assumptions on structure and discrepancies across environments will depend on the underlying causal graph and functional associations between variables.
Figure 8: Regret under misspecification.
Most of the discussion in the main body of this paper has assumed knowledge of a discrepancy between environments without considering its magnitude, although the previous example shows that knowledge of bounds or distributions of some of the variables that differ in the target domain can be incorporated. It is worth noting that other kinds of domain knowledge may be available, such as in the functional form of an association in the target domain, specific magnitudes of differences expressed within such a functional form, etc. It would be an interesting task to consider how to use that information optimally.
B.2 Run time evaluations
Reward distributions  $P^{*}(y_{x})$  induced by any underlying SCM can be expressed by a corresponding discrete SCM in which  $P(\mathbf{U})$  is discrete and  $\mathcal{F}$  is a discrete mapping between finite spaces. The complexity of the corresponding parameterization depends on the underlying selection diagram and dimensionality of observed variables.
In particular, in a given iteration of the Gibbs sampler, posterior updates are done for each parameter separately so that computational time is proportional to the parameter count, approximately, which in turn is determined by the cardinality of variables as well as the structure of the graph. For a fixed graph, assuming that each update requires a small constant amount of time to compute, we could therefore establish analytically how computational time scales with the cardinality of variables. For arbitrary graphs, an analytical result is in general more involved as the parameter count increases differently depending on the local structure of each variable. As an example for illustration, consider the graph in Fig. 9a where  $X$  is an action variable,  $Z$  is a contextual variable, and  $Y$  is a reward variable. Following the parameterization in Corol. 1, the cardinality of parameters is defined as follows:  $|\pmb{\theta}_U| = |\Omega_X| \cdot |\Omega_Z| \cdot |\Omega_Y|$ ,  $|\pmb{\xi}_X| = |\Omega_X| \cdot |\Omega_Z| \cdot |\Omega_Y|$ ,  $|\pmb{\xi}_Z| = |\Omega_X|$ ,  $|\pmb{\xi}_Y| = |\Omega_X| \cdot |\Omega_Z| \cdot |\Omega_Y| \cdot |\Omega_Z|$ .
We would expect run time to increase linearly with the cardinality of variables  $X, Y$  and to increase "slower than quadratically" with the cardinality of  $Z$ . Fig. 9b gives run time evaluations of the proposed bandit algorithm using 1000 rounds of experimentation as a function of the number of actions. Fig. 9c evaluates the influence of the size of prior data on run time by plotting the time required for drawing 1000 samples from the prior  $P(\cdot \mid \vec{v})$  with increasing prior data size (in addition to 1000 samples Gibbs samples that are discarded as burn-in). For both Figs. 9b and 9c we observe run time to scale approximately linearly with the cardinality of the action space and sample size.
C Theoretical results
Counterfactuals and optimal actions We have defined in Sec. 1.1 the fact that intervening in an SCM results in a new model that differs in the causal mechanisms subject to interventions. In this section, we extend this notion to consider models derived from interventions that replace a causal mechanism with another function, not necessarily a constant. This gives consistent definitions for counterfactual random variables of the form  $Y_{X}$  that appear in Sec. 3.
Definition 4 (Intervened model). Let  $\mathcal{M}$  be an SCM,  $\hat{\pmb{U}}\subseteq \pmb{U}$ ,  $X\in V$ , and  $\hat{X}:\hat{U}\to \Omega_X$  a function. Then,  $\mathcal{M}_{\hat{X}}$ , called the interveneded model of  $\mathcal{M}$  subject to  $\hat{X}$ , is identical to  $\mathcal{M}$ , except that the function  $f_{X}$  is replaced with the function  $\hat{X}$ .
Uncertainty in  $\mathcal{M}_{\hat{X}}$  is similarly encoded by the distribution  $P(\boldsymbol{U})$  which when averaged over leads to random variables  $Y_{\hat{X}}$  for  $Y \in V$  that describes the random variable  $Y$  under a model  $\mathcal{M}_{\hat{X}}$ . This definition is tightly related to that of potential outcome as defined in Sec. 1.1, but the former explicitly allows for interventions that do not necessarily fix the variable  $X$  to a constant value. Probabilities in this model can be computed using the chain rule
$$
P \left(Y _ {X} = y\right) = \int_ {\Omega_ {X}} P \left(Y _ {x} = y \mid X = x\right) P (X = x) d x. \tag {13}
$$
And similarly expectations are given by
$$
\mathbb {E} _ {P} \left[ Y _ {X} \right] = \int_ {\Omega_ {Y}} \int_ {\Omega_ {X}} y P \left(Y _ {x} = y \mid X = x\right) P (X = x) d x d y. \tag {14}
$$
In turn, which action is optimal is represented by a random variable  $\tilde{X}:\Omega_{\pmb{\xi}}\times \Omega_{\pmb{\theta}}\mapsto \Omega_{X}$  where  $\Omega_{\pmb{\xi}}\times \Omega_{\pmb{\theta}}$  defines the space of possible models  $M\coloneqq M(\pmb {\xi},\pmb {\theta})\in \mathcal{M}(\mathcal{G}_{\pi^{*}})$  that parameterize the deployment environment  $\pi^{*}$ . For example,  $P_{M\sim \mathcal{M}(\mathcal{G}_{\pi^*})}(\tilde{X} = x) = P_{M\sim \mathcal{M}(\mathcal{G}_{\pi^*})}(\mathbb{E}_{P_M}[Y_x]>\mathbb{E}_{P_M}[Y_{x'}],\forall x'\in \Omega_X\backslash \{x\})$  where  $P_{M\sim \mathcal{M}(\mathcal{G}_{\pi^*})}$  is a probability mass function defined over the space of possible models  $\mathcal{M}(\mathcal{G}_{\pi^*})$ .
Information Theory Definitions This section provides a definition of all the information-theoretic terms used throughout and shows that they can be generalized to mixed quantities that include discrete and continuous random variables in a consistent manner. For a discrete random variable  $X \in \Omega_X$ , the Shannon entropy and its conditional counter parts are defined as
$$
\mathcal {H} (X) := - \sum_ {x \in \Omega_ {X}} P (X = x) \log P (X = x),
$$
$$
\mathcal {H} (X \mid Z = z) := - \sum_ {x \in \Omega_ {X}} P (X = x \mid Z = z) \log P (X = x \mid Z = z) \text {a n d},
$$
$$
\mathcal {H} (X \mid Z) := \sum_ {z \in \Omega_ {Z}} \mathcal {H} (X \mid Z = z) P (Z = z).
$$
With a slight abuse of notation, we will define the filtered entropy as
$$
\mathcal {H} (X \mid \bar {\boldsymbol {v}}) := - \sum_ {x \in \mathcal {X}} P (X = x \mid \bar {\boldsymbol {v}}) \log P (X = x \mid \bar {\boldsymbol {v}}),
$$
where  $\bar{v}$  is a dataset.
Given two probability measures  $P$  and  $Q$ , where  $P$  is absolutely continuous w.r.t.  $Q$  so that the Radon-Nikodym derivative  $\frac{dP}{dQ}$  is well defined, the Kullback-Leibler divergence is
$$
\mathcal {D} _ {K L} (P \| Q) := \int \log \frac {d P}{d Q} d P,
$$
which allows the standard definition of mutual information:
$$
I _ {P} (X; Y) := \mathcal {D} _ {K L} (P (X, Y) \| P (X) P (Y)),
$$
where the subscript  $P$  that denotes the probability distribution used is sometimes omitted if unambiguous. Equivalently, we can define the mutual information and conditional mutual information by
$$
I (X; Y) = \mathcal {H} (X) - \mathcal {H} (X \mid Y) \text {a n d} I (X; Y \mid Z) = \mathcal {H} (X \mid Z) - \mathcal {H} (X \mid Y, Z),
$$
and we will also need the filtered mutual information,
$$
I _ {P (\cdot | Z = z)} (X; Y) := \mathcal {D} _ {K L} (P (X, Y \mid Z = z) \| P (X \mid z) P (Y \mid Z = z)),
$$
where we will typically evaluate  $I_{P(\cdot|\bar{V}_t,\bar{v})}(X;Y) \coloneqq D(P(X,Y\mid \bar{V}_t,\bar{v})\| P(X\mid \bar{V}_t,\bar{v})P(Y\mid \bar{V}_t,\bar{v}))$  where recall  $\bar{V}_t \coloneqq \{V_{X^{(1)}},\dots ,V_{X^{(t - 1)}}\}$  denotes the random variables associated with the agent's history of interactions with  $\pi^*$  up to round  $t$ . The conditional mutual information is related to the filtered mutual information by an expectation:  $\mathbb{E}[I_{P(\cdot|Z)}(X;Y)] = I(X;Y\mid Z)$ , with respect to  $P(Z)$ . We will frequently use this fact to show that
$$
\mathbb {E} \left[ I _ {P (\cdot | \bar {\boldsymbol {V}} _ {t}, \boldsymbol {v})} (X; Y) \right] = I _ {P (\cdot | \boldsymbol {v})} (X; Y \mid \bar {\boldsymbol {V}} _ {t}),
$$
where the expectation is taken with respect to  $P(\bar{V}_t \mid \bar{v})$ . Perhaps the most important property of mutual information will be the chain rule,
$$
I (X; Z _ {1}, \dots , Z _ {m}) = I (X; Z _ {1}) + I (X; Z _ {2} \mid Z _ {1}) + \dots + I (X; Z _ {m} \mid Z _ {1}, \dots , Z _ {m - 1}).
$$
We will need to evaluate  $I(\pmb{\theta}, \pmb{\xi}; Z)$ , where  $Z$  is a discrete random variable, e.g.  $\bar{v}_t$  or  $\pmb{V}^{(t)}$ ,  $\pmb{\theta}$  is continuous, and  $\pmb{\xi}$  is discrete. We can evaluate this mutual information by writing
$$
\begin{array}{l} I (\boldsymbol {\theta}, \boldsymbol {\xi}; Z) = I (Z; \boldsymbol {\theta}, \boldsymbol {\xi}) \\ = I (Z; \boldsymbol {\theta}) + I (Z; \boldsymbol {\xi} \mid \boldsymbol {\theta}) \\ = \mathcal {H} (Z) - \mathcal {H} (Z \mid \boldsymbol {\theta}) + \mathcal {H} (Z \mid \boldsymbol {\theta}) - \mathcal {H} (Z \mid \boldsymbol {\xi}, \boldsymbol {\theta}), \\ \end{array}
$$
which are all well defined.
For readability, we restate all propositions.
Proposition 1 (restated). Let  $R_T$  denote the regret incurred by following Thompson sampling (Alg. 1). For any  $T \in \mathbb{N}$  and  $\Gamma \geqslant \Gamma_t$ , then
$$
\mathbb {E} \left[ R _ {T} \mid \bar {\boldsymbol {v}} \right] \leqslant \Gamma \sqrt {\mathcal {H} (\tilde {X} \mid \bar {\boldsymbol {v}}) T},
$$
Proof. This analysis extends [35, Prop. 1] to account for the case in which prior data is conditioned upon. We first bound the expected regret by using the law of total expectations and introducing the mutual information between the optimal action and the action observation tuple,
$$
\begin{array}{l} \mathbb {E} \left[ R _ {T} \mid \bar {\boldsymbol {v}} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} Y _ {\tilde {X}} - Y _ {X ^ {(t)}} \mid \bar {\boldsymbol {v}} \right] (15) \\ = \mathbb {E} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ Y _ {\tilde {X}} - Y _ {X ^ {(t)}} \mid \bar {\boldsymbol {V}} _ {t}, \bar {\boldsymbol {v}} \right] (16) \\ = \mathbb {E} \sum_ {t = 1} ^ {T} \sqrt {I _ {P (\cdot | \bar {V} _ {t} , \bar {\boldsymbol {v}})} (\tilde {X} ; \boldsymbol {V} _ {X ^ {(t)}})} \frac {\mathbb {E} \left[ Y _ {\tilde {X}} - Y _ {X ^ {(t)}} \mid \bar {V} _ {t} , \bar {\boldsymbol {v}} \right]}{\sqrt {I _ {P (\cdot | \bar {V} _ {t} , \bar {\boldsymbol {v}})} (\tilde {X} ; \boldsymbol {V} _ {X ^ {(t)}})}}. (17) \\ \end{array}
$$
By using the KL divergence definition of mutual information and using the fact that Thompson sampling precisely chooses actions according to their probability of being optimal, it can be shown that,
$$
\begin{array}{l} \mathbb {E} \left[ Y _ {\tilde {X}} - Y _ {X ^ {(t)}} \right] = \sum_ {x \in \Omega_ {X}} P (\tilde {X} = x) \mathbb {E} \left[ Y _ {x} \mid \tilde {X} = x \right] - \sum_ {x \in \Omega_ {X}} P (X = x) \mathbb {E} \left[ Y _ {x} \right] \\ = \sum_ {x \in \Omega_ {X}} P (\tilde {X} = x) (\mathbb {E} [ Y _ {x} \mid \tilde {X} = x ] - \mathbb {E} [ Y _ {x} ]), \\ \end{array}
$$
where  $P(X = x) = P(\tilde{X} = x)$  by definition of the policy used by Thompson sampling. Let the information ratio  $\Gamma_t$  is defined as an upperbound on the ratio