Title: Transportability for Bandits with Data from Different Environments
Venue: NeurIPS 2023
Accepted: yes
Keywords: Transportability

transfer learning

bandits
Abstract: A unifying theme in the design of intelligent agents is to efficiently optimize a policy based on what prior knowledge of the problem is available and what actions can be taken to learn more about it. Bandits are a canonical instance of this task that has been intensely studied in the literature. Most methods, however, typically rely solely on an agent's experimentation in a single environment (or multiple closely related environments). In this paper, we relax this assumption and consider the design of bandit algorithms from a combination of batch data and qualitative assumptions about the relatedness across different environments, represented in the form of causal models. In particular, we show that it is possible to exploit invariances across environments, wherever they may occur in the underlying causal model, to consistently improve learning. The resulting bandit algorithm has a sub-linear regret bound with an explicit dependency on a term that captures how informative related environments are for the task at hand; and may have substantially lower regret than experimentation-only bandit instances.

Transportability for Bandits with Data from Different Environments

Abstract

A unifying theme in the design of intelligent agents is to efficiently optimize a policy based on what prior knowledge of the problem is available and what actions can be taken to learn more about it. Bandits are a canonical instance of this task that has been intensely studied in the literature. Most methods, however, typically rely solely on an agent's experimentation in a single environment (or multiple closely related environments). In this paper, we relax this assumption and consider the design of bandit algorithms from a combination of batch data and qualitative assumptions about the relatedness across different environments, represented in the form of causal models. In particular, we show that it is possible to exploit invariances across environments, wherever they may occur in the underlying causal model, to consistently improve learning. The resulting bandit algorithm has a sub-linear regret bound with an explicit dependency on a term that captures how informative related environments are for the task at hand; and may have substantially lower regret than experimentation-only bandit instances.

1 Introduction

Multi-armed bandits (MABs) constitute one of the most widely used frameworks for modeling decision-making under uncertainty. In this framework, an agent repeatedly takes actions in an environment with the goal of optimizing a desired objective, such as efficiently inferring the action with highest reward or maximizing cumulative rewards in the long run [34]. As in most reinforcement learning problems, there is a substantial amount of exploration involved while the agent learns about reward distributions under different available actions. This process can be costly in many applications; from an ethical perspective, for example, physicians may not risk compromising their patient's health with unknown treatments. It is therefore important to be efficient with experimentation while learning an optimal policy. In the literature, structured bandit instances can help navigate the explorationexploitation trade-off effectively, for example with assumptions on the functional association between action and reward that facilitate estimation such as linear bandits [1, 15, 19] and causal bandits [23, 36, 28, 27, 24, 26, 12, 29, 7].

An alternative approach to alleviate the cost of active experimentation is to consider leveraging prior data or prior experimentation in related environments to inform an agent's decision-making, which leads to the hybrid learning paradigm. The expectation (or rather hope) is that informative prior data or prior experimentation can serve to narrow down reward distributions and warm start the MAB so as to converge to optimal actions faster and ultimately achieve higher cumulative reward. Current methods can be categorized into multi-task learning [41, 40, 13] and meta-learning [10, 42, 4, 20, 33, 30]. The former aims to solve a prescribed set of related bandit tasks with shared structure, e.g. multiple player scenarios with similar reward distributions. The latter considers an arbitrary number of bandit problems whose parameters are sampled i.i.d. from a meta-prior that

can be inferred as the agent experiments across the different related tasks 1 . However, both families of methods assume a relatively restricted class of potential changes across environments and rely explicitly on agent experimentation across all environments for learning. If discrepancies across environments are more general, naively leveraging prior data does not necessarily lead to more informative reward distributions or efficiency improvements in a new environment.

As a concrete example, consider a learning scenario in which historical data is available for the design of a clinical trial 2 aiming to determine the optimal level of a hypertension treatment X for Alzheimer's disease Y 3 . Alzheimer's aetiology is complex but it is well established that a patient's age Z and blood pressure W contribute to the development of the disease, and so do a number of (typically) unobserved factors, e.g. , physical activity levels, socio-economic status, diet patterns, etc. [37] (encoded with a bi-directed dashed edge). Such data can be useful but has to be handled with care, especially if we suspect the clinical trial popula-

X

W

Y

Z

S

Z

tion to differ in several aspects from that recorded in historical data. We would expect, for example, age distributions P p z q to differ. Fig. 1 graphically describes this scenario. A naive approach, ignoring the differences across populations, can be sub-optimal. Take an instance where variables X,Z,W,Y,U P t 0 , 1 u ; their values are decided by functions: X Ð U, W Ð X,Y Ð Z 'p W ¨ U q ; ' represents the exclusive-or operator; U is independently distributed in t 0 , 1 u but older individuals are historically over-represented P p Z ' 1 q ' 0 . 6 in comparison to the clinical trial population P p Z ' 1 q ' 0 . 4 . Historical data suggests that the better policy involves a lower dosage X ' 0 , as E r Y | do p X ' 0 qs ' 0 . 6 ą 0 . 5 ' E r Y | do p X ' 1 qs , which is the opposite of what is optimal for the clinical trial population.

This example shows that differences across environments may be complex, subtle, and non-trivially influence optimal decision-making. Even when one is able to perfectly estimate reward distributions from historical data, the induced policy can still be sub-optimal depending on the location and magnitude of the changes expected across environments. In general, reward distributions will not straightforwardly extrapolate across different environments. In this paper, we attempt to capture this (structural) uncertainty through a causal lens. In the causality literature, this problem appears under the rubric of transportability theory [32, 3, 11]; several criteria, algorithms, and estimation methods have been developed for identifying when and how a causal effect can be computed across environments. Our task, in the bandit setting, is to define a learning agent that optimally exploits prior data with knowledge of the potential discrepancies across domains, for any given graph. Prior work [43] has considered specific instances, i.e. environments defined by the Bow and Instrumental Variables graphs, but a general approach applicable to arbitrary graphs and arbitrary differences across environments is still missing. Our approach is Bayesian, and involves posterior sampling of reward distributions defined by a parameterization informed by the underlying causal structure. Our contribution is to develop a novel bandit algorithm that achieves sub-linear cumulative regret with an explicit dependency on the entropy of an inferred prior, a quantity that implicitly captures the relatedness between environments. The significance of this result is that it guarantees consistent improvements on performance over methods not leveraging prior data. To the best of our knowledge, this is one of the first general attempts to consistently use prior data from related environments in general decision-making scenarios in which causal dependencies can be established.

1.1 Preliminaries

We use capital and small letters to denote random variables and their values respectively, e.g. X and x , and bold capital and small letters to denote sets of variables and their values, e.g. X and x . The domain of variable X is indicated with Ω X .

A environment's data generating mechanism is described by a structural causal model (SCM) [31, Definition 7.1.1]. A SCM M is a tuple x V , U , F , P p U qy , where V is a set of endogenous (observed) variables, U is a set of exogenous latent variables, and F ' t f V u V P V is a set of functions

such that f V determines values of V taking as argument variables Pa V Ď V and U V Ď U , i.e. V Ð f V p Pa V , U V q . Values of U are drawn from an exogenous distribution P p u q . We assume the model to be recursive, i.e. that there are no cyclic dependencies among the variables, such as to define a distribution P p V q over endogenous variables V . An intervention or action by an agent on a subset X Ă V , denoted by do p x q , is an operation that fixed values of X to constants x , replacing the functions t f X : X P X u that would normally determine their values. Let M x denote the model induced by action do p x q . Accordingly, M x induces a corresponding interventional distribution over V , denoted P p V x q : ' P p V | do p x qq . We will consistently use X,Y P V as designated action and reward variables, respectively.

Causal graphs G ' p V , E q describe the functional associations in an underlying SCM M . In particular, we draw a directed edge between two variables V Ñ W P E if V appears as an argument of f W in M , and a bi-directed dashed edge between two variables V glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] W P E if U V X U W ‰ H , i.e. V and W share an unobserved confounder . We will use standard family conventions for graphical relationships, e.g. parents pa p X q G : ' Y X P X pa p X q G of a set of nodes X Ď V are all nodes in G with directed edges into elements of X . Its capitalized version Pa includes the argument as well, e.g. Pa p X q G : ' pa p X q G Ť X . We will make use a special clustering of the nodes in V called c -components [39]: two nodes are in the same c -component C Ď V if and only if they are connected by a bi-directed path. c -components form a partition over exogenous variables: a c -component C Ď V is said to cover an exogenous variable U if U P Ť V P C U V . We denote with C U the c -component covering U . As an example, the diagram in Fig. 1 has c -components t X,Y u , t W u , and t Z u ; and C U XY ' t X,Y u , C U W ' t W u , and C U Z ' t Z u . We refer the reader to [31, Chapter 7] for a more detailed review of SCMs.

2 Bandits with Transportability

From the agent's perspective, the point of departure with respect to conventional bandit instances is that in addition to the ability to take actions in a deployment environment π ˚ , the agent has access to data from one or more related environments π a , π b , . . . , each characterized by SCMs M a , M b , . . . . We assume that all environments have the same scope, i.e. the same sets V and U , but may differ in any other aspect. In the transportability literature [3, 32], such structural differences between environments are called domain discrepancies and can be encoded in selection diagrams.

Definition 1 (Domain Discrepancy) . Let π a and π b be two domains with SCMs M a and M b . There exists a domain discrepancy between π a and π b if f a V ‰ f b V or P a p U V q ‰ P b p U V q for some V P V .

Definition 2 (Selection diagram) . Given domain discrepancy set ∆ a,b : ' t V P V : f a V ‰ f b V or P a p U V q ‰ P b p U V qu between two domains π a and π b and a causal graph G a ' p V , E q , let S ' t S V : V P ∆ a,b u be called selection nodes. The graph G a,b ' p V Y S , E Yt S V Ñ V u S V P S q is called selection diagram.

Selection nodes indicate where structural discrepancies between two environments might take place. The absence of a selection node pointing to a variable represents the assumption that the causal mechanism responsible for assigning values to that variable is identical in both environments. In the clinical trial example, Fig. 1 shows a selection diagram comparing historical and clinical trial environments, denoted π ˚ , π a respectively; the presence of selection node S Z indicates a potential difference in the assignment of Z , i.e. , either f ˚ Z ‰ f a Z and / or P ˚ p u W q ‰ P a p u W q . On the other hand, the absence of e.g. selection node S Y indicates the assumption f ˚ Y ' f a Y and P ˚ p u Y q ' P a p u Y q . With this formalism, the task is to leverage data from related environments in a consistent and efficient manner.

Definition 3 (Bandits with Transportability) . Let π ˚ denote the deployment environment in which the agent acts. Given samples from P a p V q , P b p V q , . . . and selection diagrams G ˚ ,a , G ˚ ,b , . . . , in each round t ' 1 , . . . , T the agent takes an action x p t q and observes a sample from P ˚ p V x p t q q , adjusting its actions to minimize (expected) cumulative regret in π ˚ ,

that compares the optimal intervention ˜ x ' arg max x P Ω X E P ˚ Y x with the agent's chosen intervention in each round.

Quantities such as E P ˚ Y x or P ˚ p y x q are called transportability queries and their estimation with prior data, underlying structural assumptions and their use within active experimentation schemes will be the focus of this paper.

2.1 Informative priors for bandits

Before experimentation takes place there is a degree of unidentifiability of reward distributions P ˚ p y x q depending on causal assumptions and discrepancies between environments. For instance, revisiting the clinical trial example, if age distributions are allowed to vary arbitrarily across environments, values of P ˚ p y x q will similarly vary and thus involve a degree of uncertainty 4 . This unidentifiability feature is relevant even without major discrepancies across domains as P ˚ p y x q may still be unidentifiable in the presence of unobserved confounders. The Bow graph in Fig. 2 (ignoring the selection node) is a common example.

One may be tempted to conclude that prior data is rarely useful. However, even under multiple discrepancies across environments, causal effects P ˚ p y x q are rarely completely unconstrained. In general, causal effects lie in a non-trivial interval r a, b s , 0 ă a ď b ă 1 . For instance, for the graph G a, ˚ in Fig. 2 P ˚ p y x q can be shown to be contained in r P a p x, y q , P a p x, y q ` 1 ´ P a p x qs . In particular, with a probabilistic or Bayesian interpretation of unknown quantities in a SCM, that is with an explicit probability measure over SCMs M P M p G ˚ q 5 , one

G

a,

˚

X

Y

S

X

can define a distribution of reward probabilities P ˚ p y x q (or expected rewards) that honestly captures prior uncertainty in their values. For example, given prior samples ¯ v a ' t v a p j q : j ' 1 , . . . , 500 u independently drawn from a source distribution P a p V q (and a particular prior model over SCMs, to be discussed in Sec. 2.2), a posterior density over E P ˚ Y x can be evaluated and sampled from, see Fig. 2 for an illustration, where theoretical bounds are shown with vertical lines.

From this perspective, the problem of doing inference on reward distributions given prior data and knowledge of structural discrepancies can be formulated as an optimization problem, that is, evaluate,

where ¯ v : ' t ¯ v a , ¯ v b , . . . u , ¯ v i ' t v i p j q : j ' 1 , . . . , n i u is a set of n i independent samples from P i p V q . In words, the task is to evaluate a distribution over expected rewards under intervention over all deployment domains π ˚ compatible with our knowledge prior to experimenting in π ˚ .

It remains a question, however, how to define a model and parameterization of SCMs. Remember that only prior data and selection diagrams are assumed to be available to the researcher; any choices on the distribution of exogenous variables P p U q or functional form of deterministic structural assignments F represent untestable assumptions that are difficult to justify in practice. Going forward we will restrict ourselves to SCMs with discrete endogenous variables (while exogenous variables may be arbitrarily defined, e.g. continuously-valued with arbitrary probability density functions).

2.2 General parameterization of reward distributions

Systems of discrete observables have the distinctiveness of involving a finite number of probabilities of the form P ˚ p y x q , i.e. one for each combination p x, y q . Reward distributions P ˚ p y x q in any underlying SCM M , however complex P p U q and F may be, can logically be equivalently expressed by a corresponding discrete SCM N in which P p U q is discrete and F is a discrete mapping between finite spaces [44, 6]. This observation is interesting because it allows us to consistently and uniquely parameterize P p U q and F , without untestable choices on their form 6 .

Corollary 1 (Proposition 2.7. [44]) . For any causal graph G , let M be an arbitrary SCM compatible with G . For any sets Y , X Ă V , the interventional distribution P p y x q could be parameterized as

where θ u : ' P p U ' u q defines exogenous probabilities of discrete variables U P U with cardinality d U ' ś V P C U ˇ ˇ Ω Pa p V q ˇ ˇ ; and each ξ p pa V , u V q V is a deterministic mapping between finite domains Ω Pa V ˆ Ω U V ÞÑ Ω V .

For example, P ˚ p y x q in Fig. 1 can be parameterized by

where, assuming X,Y,Z,W are binary, θ u z is a discrete distribution over a finite domain t 1 , 2 u since | Ω U z | ' | Ω Pa p Z q | ' | Ω Z | ' 2 , θ u w is a distribution over t 1 , . . . , 4 u since | Ω U W | ' | Ω Pa p W q | ' | Ω X | ¨ | Ω Z | ' 4 , and θ u xy is distribution over t 1 , . . . , 32 u since | Ω U XY | ' | Ω Pa p X q | ¨ | Ω Pa p Y q | ' | Ω Z | ¨ | Ω X | ¨ | Ω W | ¨ | Ω Z | ¨ | Ω Y | ' 32 . Corol. 1 guarantees that for any value of P ˚ p y x q induced by an arbitrary SCM M there exists a combination of parameters in Eq. (4) that reaches that exact same value. In other words, this parameterization is sufficiently expressive to encode any underlying reward distribution.

Parameters that define reward distributions are specific to the deployment environment π ˚ . The key observation, however, is that some of them can be inferred with prior data whenever there exists an invariance in P p U q or F across environments as there exists a one-to-one relationship between model parameters and structural features of the underlying SCM. For example, given Fig. 1, the absence of a selection node into Y implies that both functional assignments and exogenous probabilities of Y agree across environments, that is ξ ˚ Y ' ξ a Y , θ ˚ u Y ' θ a u Y 7 . Both may thus be approximated with prior data which in turn constraints or informs P ˚ p y x q even if other parameters in its expression in Eq. (4) remain unknown. The result is a non-trivial distribution over reward probabilities that may be used to warm-start bandit algorithms, even before any experimentation takes place.

2.3 Bandit algorithms

To exploit non-trivial parameter distribution given prior data, a bandit algorithm can be designed to choose actions in proportion to the probability that an intervention leads to highest reward, also known as posterior or Thompson sampling [38, 2]. Specifically, at a particular round t of experimentation in the deployment domain π ˚ , posterior parameter distributions P p ξ , θ | ¯ v , v x p 1 q , . . . , v x p t ´ 1 q q can be evaluated to exactly capture uncertainty given both prior and experimental data up to round t . Action x p t q is then chosen according to the one that gives highest reward, i.e. arg max x E P ˚ ' Y x | ξ p t q , θ p t q ‰ , where p ξ p t q , θ p t q q is an independent draw from its posterior distribution. In other words, the agent

performs natural Bayesian updates based on both the data available in source environments and its own experimentation as interventional samples v x p 1 q , . . . , v x p t ´ 1 q become available, matching the intuition of most other Thompson sampling bandits in the literature. The full algorithm, called Thompson sampling with Transportability ( tTS ), is given in Alg. 1.

3 Regret guarantees conditional on prior data

We define information-theoretic regret bounds that aim to capture the exploration-exploitation tradeoff for Alg. 1 when prior information allows it to infer parts of the environment before experimentation takes place.

Performance in MABs is, to a large extent, intimately related with the agent's uncertainty about which action is optimal, represented by a random variable ˜ X : Ω ξ ˆ Ω θ ÞÑ Ω X where Ω ξ ˆ Ω θ defines the space of all models M p G ˚ q consistent with our knowledge of the deployment environment π ˚ . For example, P M ' M p G ˚ q p ˜ X ' x q ' P M ' M p G ˚ q p E P M r Y x s ą E P M r Y x 1 s , @ x 1 P Ω X zt x uq where P M ' M p G ˚ q is a probability mass function defined over M p G ˚ q . It is reasonable to assume that one would only choose actions with large regret when it can reduce the uncertainty in ˜ X substantially. Following [35, Sec. 5], we define a scalar Γ t 8 such that the per-round regret can be bounded by information gain,

where ¯ V t : ' t V X p 1 q , . . . , V X p t ´ 1 q u denotes the agent's history of interactions with π ˚ up to round t , and I P p X,Y q : ' D KL p P p X,Y q} P p X q P p Y qq denotes the filtered mutual information defined based on P (where D KL is the Kullback-Leibler divergence). Expectations, unless otherwise stated, are taken with respect to all random quantities. The following proposition, extended from [35, Prop. 1], shows that the Bayesian regret of an agent acting according to Alg. 1 is sub-linear with a dependency on the entropy of the optimal action ˜ X .

Proposition 1. Let R T denote the regret incurred by following Thompson sampling (Alg. 1). For any T P N and Γ ě Γ t , then

where H p ˜ X | ¯ v q is the conditional entropy of ˜ X given ¯ v .

This bound is interesting because it cleanly relates the regret with the uncertainty about the optimal action conditioned on prior data. On one extreme, if data from source environments fully characterizes the optimal action, the entropy equals 0, and no further experimentation is required; on the other extreme, if data from source environments have no relationship with the target query, the entropy equals log p| Ω X |q , and the bound reverts to conventional worst-case guarantees [35]. The entropy of the optimal action is often not sufficient to capture the information from ¯ v as ˜ X may still have a uniform distribution even though posterior distributions over p θ , ξ q have tightened. In other words, there is additional structure among different reward distributions that is not captured by the entropy of the optimal action. Such a setting can be analyzed with a different assumption on the per round regret that explicitly considers model parameters p θ , ξ q to quantify information gain,

where ϵ t ą 0 is an additional slack term. Accordingly, the following proposition provides an alternative bound using a conditional analogue of [25, Prop. 2].

Proposition 2. Let R T denote the regret incurred following the policy defined by Alg. 1. For any T P N , if Eq. (7) holds with Γ ě Γ t for all t ,

This proposition shows that if prior data allows the agent to concentrate p θ , ξ q around some value, additional experimentation does not provide much more information and the regret should be small. In principle, it is possible to get precise per-round regret values by inferring values for Γ t and ϵ t through the construction of concentration inequalities for the reward variable, as done by Lu et al. in [25, Lem. 3]. We adapt this result using conditional information-theoretic quantities in the following proposition.

Proposition 3. Fix δ ą 0 and choose Γ t such that ˇ ˇ Y x ´ E r Y x | ¯ V t , ¯ v s ˇ ˇ ď Γ t 2 b I P p¨| ¯ V t , ¯ v q p θ , ξ ; Y x q for all x P Ω X simultaneously with probability greater than 1 ´ δ . Then Alg. 1 chooses actions X p t q that satisfy

where B ě 0 is such that sup y,y 1 P Ω Y y ´ y 1 ď B .

4 Posterior approximations

This section describes a tractable algorithm to evaluate posterior distributions of the form P p ξ , θ | ¯ v , v x p 1 q , . . . , v x p t ´ 1 q q and its posterior updates when new data is collected. Priors on ξ , θ may be defined such as to induce tractable conditional distributions that may be used within a Gibbs sampling framework. The Gibbs sampler starts with some initial value for all latent quantities p U , ξ , θ q in our target expected reward

and samples each one iteratively using their conditional distributions, each parameter conditioned on the current values of the remaining terms in the parameter vector and the available data [16]. As mentioned, what data point carries information about which parameters depends on the structural differences between environments.

Prior. For every V P V , @ pa V , u V , the functional assignment parameters ξ p pa V , u V q V are drawn uniformly in the discrete domain Ω V . For every U P U , exogenous probabilities θ U with dimension d U ' ś V P C U ˇ ˇ Ω Pa p V q ˇ ˇ are drawn from a prior Dirichlet distribution (here chosen for conjugacy with the categorical distribution of U ), θ U ' p θ 1 , . . . , θ d U q ' Dir p α 1 , . . . , α d U q , with hyperparameters α 1 , . . . , α d U .

Posterior. In a particular round t , the Gibbs sampler iterates over the following steps.

Sample u . We start by sampling a corresponding exogenous latent variable for each observed sample v p n q P p ¯ v , v x p 1 q , . . . , v x p t ´ 1 q q , n ' 1 , . . . , t ´ 1 ` ř i n i . Let p ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q denote the corresponding set of samples of U . Exogenous variables U p n q are mutually independent given V p n q , ξ , θ and thus we can sample each separately using the conditional

Sample ξ . Similarly, for fixed pa V , u V , parameters ξ p pa V , u V q V are mutually independent given ¯ v , v x p 1 q , . . . , v x p t ´ 1 q , ¯ u , u x p 1 q , . . . , u x p t ´ 1 q , θ . As mentioned, each parameter is updated with the subset of p ¯ v , v x p 1 q , . . . , v x p t ´ 1 q , ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q associated with environments in which the functional assignment of V is invariant across source and deployment environments. As they represent a mapping between variables, its conditional distribution is given by P p ξ p pa V , u V q V ' v | ¯ v , v x p 1 q , . . . , v x p t ´ 1 q , ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q ' 1 if there exists a (relevant) sample p v p n q , pa p n q V , u p n q V q that fixes the mapping pa p n q V , u p n q V ÞÑ v p n q . Otherwise, P p ξ p pa V , u V q V ' v | ¯ v , v x p 1 q , . . . , v x p t ´ 1 q , ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q is sampled uniformly from a discrete distribution over Ω V .

Sample θ . Fix U P U . By conjugacy of Dirichlet distributions with the categorical distribution, its conditional distribution given all other quantities is given by a Dirichlet distribution θ U |

X

Y

Z

S

Z

(a)

G

a,

˚

(b) Regret without prior data.

(c) Regret with prior data.

P(- I Vx(),...,Vx)

1.0

Expected rewards

Actionx=0

0.8

Actionx=1

0.6

0.4

0.2

0.0

0

100

200

300

400

500

Rounds

P(·|v)

P(·|v,Vx(1）,...,Vx①)

1.0

rewards

Actionx=0

0.8

Action x=1

0.6

Expectedr

0.4

0.2

0.0

口

0

100200300

0400500

Rounds

¯ v , v x p 1 q , . . . , v x p t ´ 1 q , ¯ u , u x p 1 q , . . . , u x p t ´ 1 q ' Dir p β 1 , . . . , β d U q where β j : ' α j ` ř n ✶ t u p n q ' u j u , and, similarly, n iterates over the samples p ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q associated with the subset of environments in which exogenous probabilities match the deployment environment.

This procedure eventually forms a Markov chain with the invariant distribution P p u , ξ , θ | ¯ v , ¯ v x p 1 q , . . . , ¯ v x p t ´ 1 q q . We plug in one of these samples ξ , θ ' P p ξ , θ | ¯ v , ¯ v x p 1 q , . . . , ¯ v x p t ´ 1 q q into Eq. (10) for different x to choose the next action x p t q . This sample initializes the chain for P p u , ξ , θ | ¯ v , ¯ v x p 1 q , . . . , ¯ v x p t q q in round t of Alg. 1.

So far, we have described algorithms, approximations, and regret guarantees that pre-suppose the correct specification of causal and selection diagrams across multiple domains. Some degree of misspecification, however, can be tolerated without voiding guarantees on performance improvements. We discuss more details in Appendix B.1.

5 Experiments

We evaluate the proposed approach on several synthetic scenarios inspired by the literature on clinical trials and advertising. We compare Thompson sampling with additional data sources ( tTS , Alg. 1) with Thompson sampling with uninformative priors ( TS ) [38], a KL-UCB [9] algorithm with uninformative priors ( UCB ), and as a baseline also include the algorithm that chooses actions uniformly at random ( Uniform ) 9 . For all algorithms, we measure their regrets R T , averaged over 10 repetitions. Details on all data generating mechanisms and a discussion on mis-specification and limitations of the proposed approach can be found in Appendix D and Appendix B, respectively.

Experiment 1. We start by evaluating the usefulness of prior data by comparing learned distributions of expected reward with and without access to prior data. We consider a bandit problem with action, reward and contextual variables X,Y,Z P t 0 , 1 u , respectively, characterized by Fig. 3a in which 1000 prior data samples are given from an environment π a that differs in the causal assignment of Z in comparison with the deployment environment π ˚ . Specifically, with this model,

where p ξ Y , θ u xy q are invariant across environments while p ξ Z , θ u z q are specific to the deployment environment. We start by considering Fig. 3b that gives samples from E P ˚ r Y x s | v x p 1 q , . . . , v x p t q as a function of experimentation rounds t , that is without making use of prior data ¯ v . Distributions of expected rewards under action x ' 0 and x ' 1 overlap substantially until round t ' 300 at which point x ' 1 is inferred to lead to higher expected rewards. Fig. 3c gives a similar plot with the exception that the left part of the plot gives prior samples E P ˚ r Y x s | ¯ v illustrating the shape of the expected reward distribution learned from prior data only. In particular, we observe E P ˚ r Y x ' 1 s | ¯ v concentrated in the interval r 0 . 3 , 0 . 9 s and E P ˚ r Y x ' 0 s | ¯ v concentrated in the interval r 0 . 1 , 0 . 8 s . The

agent starts experimentation at round t ' 0 with this prior and from then onward expected reward samples are drawn from E P ˚ r Y x s | ¯ v , v x p 1 q , . . . , v x p t q as a function of experimentation round t . The bandit algorithm with prior data is remarkably more efficient, being able to determine x ' 1 as the superior action after only 80 rounds of experimentation. Overall, prior data leads the bandit algorithm to pull the optimal arm in 99% of time versus 93% of the time without prior data.

Experiment 2. We revisit our introductory example to quantify the benefit of leveraging historical patient data from various hospitals. In this example, the objective is to infer the optimal level of hypertension medication X . We are given a choice among 5 different levels, i.e. | Ω X | ' 5 , and wish to increase the probability of the presence of a beneficial biomarker Y , i.e. | Ω Y | ' 2 , in the clinical trial population π ˚ given that we have prior observational data in a different hospital π a . The selection diagram describing this causal protocol is given in Fig. 1. Regret comparisons for all algorithms are given in Fig. 4 (LHS). We observe a significant gain in performance by tTS that chooses the optimal intervention in 67% of the rounds in contrast with 35% of the rounds for TS , and 28% of the rounds for UCB (and 17% for an algorithm choosing interventions at random).

We use this example also to illustrate empirically the dependence between regret and prior entropy shown in Prop. 1 (RHS). For this, we consider different prior beta distributions for P ˚ p W ' 1 | x q , specifically with increasing standard deviations around the true value of P ˚ p W ' 1 | x q for each x P Ω X . A larger standard deviations implies a less informative prior and higher entropy of the random variable ˜ X that denotes the optimal action. The

300

tTS

250

TS

6

egret

200

UCB

Uniform

Regret

8

150

R

100

7

50

6

0

0

2000

4000

6000

8000

10000

5

0.0

0.2

0.4

0.6

0.8

Rounds

Square root of Entropy

entropy of ˜ X takes values in the interval r 0 . 2 , 1 . 4 s for assumed Beta priors for P ˚ p W ' 1 | x q with standard deviations in the interval r 0 . 001 , 0 . 1 s . Fig. 4 (RHS) demonstrates empirically the influence of the entropy of ˜ X on the expected cumulative regret given in Prop. 1. In particular, narrower, more informative priors lead to better regret.

Experiment 3. This example considers an advertiser seeking to optimize which ads to show visitors on a particular website. For each visitor, we choose one out of a collection of 6 ads X , | Ω X | ' 6 , some of which will be more engaging than others, to ultimately optimize whether a user clicks Y P t 0 , 1 u . Each ad has some theoretical but unknown click-through-rate P ˚ p y x q .

In this example, we assume access to 500 data points from an ad recommendation system used on a different website, i.e. a different environment π a . There, the effect of an add X on the number of clicks Y is confounded by the user's age A , (here categorized into old and young such that Ω A ' t 0 , 1 u and | Ω A | ' 2 ) and the user's product preferences level W , which interacts with current ad-recommendation system through a user's

G

a,

˚

X

Y

Z

W

A

S

A

300

tTS

250

TS

egret

200

UCB

Uniform

150

100

50

0

0

2000

4000

6000

8000

10000

Rounds

browsing history Z and location (not observed and therefore represented with a bi-directed edge between W and X ). Moreover, in this example, the relationship between W and Y is itself confounded by unobserved factors. The population visiting the website of interest π ˚ , where the MAB will be deployed, is known to agree on all causal components with π a except on the distribution of age A . This causal protocol as well as regret comparisons for this example are shown in Fig. 5. We observe noticeable improvements in regret with prior data and knowledge of structural differences as tTS substantially improves over algorithms agnostic of prior data.

6 Conclusions

This paper investigated the problem of improving the efficiency of multi-armed bandits using batch data from related environments. As source environments might differ, some knowledge of structure and (potential) discrepancies are necessary to extrapolate consistently. This paper demonstrated that knowledge of selection diagrams that encode causal influence as well as potential discrepancies

across source and target environments, without, however, an explicit specification of functional form and distributions, is sufficient to consistently define an informative prior over reward distributions using data from arbitrary environments. The resulting algorithm guarantees improvements in regret in comparison to algorithms agnostic of prior data. To our knowledge, this serves as one of the first principled approaches to consistently leverage prior data in the context of bandits and we hope it can pave the way for developing more general transfer learning methods in reinforcement learning.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning , pages 127-135. PMLR, 2013.

Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences , 113(27):7345-7352, 2016.

Soumya Basu, Branislav Kveton, Manzil Zaheer, and Csaba Szepesvári. No regrets for learning the prior in bandits. Advances in Neural Information Processing Systems , 34:28029-28041, 2021.

Alexis Bellot and Elias Bareinboim. Partial transportability for domain generalization. Technical Report R-88, Causal AI Lab, Columbia University , 2022.

Alexis Bellot, Junzhe Zhang, and Elias Bareinboim. Scores for learning discrete causal graphs with unobserved confounders. Technical Report R-83, Causal AI Lab, Columbia University , 2022.

Blair Bilodeau, Linbo Wang, and Daniel M Roy. Adaptively exploiting d-separators with causal bandits. arXiv preprint arXiv:2202.05100 , 2022.

Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research , 14(11), 2013.

Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-Leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics , pages 1516-1541, 2013.

Leonardo Cella and Massimiliano Pontil. Multi-task and meta-learning with sparse linear bandits. In Uncertainty in Artificial Intelligence , pages 1692-1702. PMLR, 2021.

Juan Correa and Elias Bareinboim. General transportability of soft interventions: Completeness results. Advances in Neural Information Processing Systems , 33:10902-10912, 2020.

Arnoud AWM de Kroon, Danielle Belgrave, and Joris M Mooij. Causal discovery for causal bandits utilizing separating sets. arXiv preprint arXiv:2009.07916 , 2020.

Aniket Anand Deshmukh, Urun Dogan, and Clay Scott. Multi-task learning for contextual bandits. Advances in neural information processing systems , 30, 2017.

Alun Evans, Veikko Salomaa, Sangita Kulathinal, Kjell Asplund, François Cambien, Marco Ferrario, Markus Perola, Leena Peltonen, Denis Shields, Hugh Tunstall-Pedoe, et al. Morgam (an international pooling of cardiovascular cohorts). International journal of epidemiology , 34(1):21-27, 2005.

Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems , 23, 2010.

Alan E Gelfand. Gibbs sampling. Journal of the American statistical Association , 95(452):13001304, 2000.

John C Gittins and David M Jones. A dynamic allocation index for the discounted multiarmed bandit problem. Biometrika , 66(3):561-565, 1979.

Limor Gultchin, Virginia Aglietti, Alexis Bellot, and Silvia Chiappa. Functional causal bayesian optimization. arXiv preprint arXiv:2306.06409 , 2023.

Botao Hao, Tor Lattimore, and Mengdi Wang. High-dimensional sparse linear bandits. Advances in Neural Information Processing Systems , 33:10753-10763, 2020.

Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical bayesian bandits. In International Conference on Artificial Intelligence and Statistics , pages 7724-7741. PMLR, 2022.

Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. Advances in neural information processing systems , 31, 2018.

Juha Karvanen. Study design in causal models. Scandinavian Journal of Statistics , 42(2):361377, 2015.

Finnian Lattimore, Tor Lattimore, and Mark D Reid. Causal bandits: Learning good interventions via causal inference. Advances in Neural Information Processing Systems , 29, 2016.

Sanghack Lee and Elias Bareinboim. Structural causal bandits with non-manipulable variables. In AAAI Conference on Artificial Intelligence , volume 33, pages 4164-4172, 2019.

Xiuyuan Lu and Benjamin Van Roy. Information-theoretic confidence bounds for reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Yangyi Lu, Amirhossein Meisami, and Ambuj Tewari. Causal bandits with unknown graph structure. arXiv preprint arXiv:2106.02988 , 2021.

Yangyi Lu, Amirhossein Meisami, Ambuj Tewari, and William Yan. Regret analysis of bandit problems with causal background knowledge. In Conference on Uncertainty in Artificial Intelligence , pages 141-150. PMLR, 2020.

Yangyi Lu, Amirhossein Meisami, Ambuj Tewari, and Zhenyu Yan. Regret analysis of causal bandit problems. arXiv preprint arXiv:1910.04938 , 2019.

Alan Malek and Silvia Chiappa. Asymptotically best causal effect identification with multiarmed bandits. Advances in Neural Information Processing Systems , 34:21960-21971, 2021.

Hyejin Park, Seiyun Shin, Kwang-Sung Jun, and Jungseul Ok. Transfer learning in bandits with latent continuity. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 1463-1468. IEEE, 2021.

Judea Pearl. Causality . Cambridge University Press, 2009.

Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal approach. In Twenty-fifth AAAI conference on artificial intelligence , 2011.

Amit Peleg, Naama Pearl, and Ron Meir. Metalearning linear bandits by prior update. In International Conference on Artificial Intelligence and Statistics , pages 2885-2926. PMLR, 2022.

Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527-535, 1952.

Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of Thompson sampling. Journal of Machine Learning Research , 17(1):2442-2471, 2016.

Rajat Sen, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Identifying best interventions through online importance sampling. In International Conference on Machine Learning , pages 3057-3066. PMLR, 2017.

Ingmar Skoog and Deborah Gustafson. Update on hypertension and alzheimer's disease. Neurological Research , 28(6):605-611, 2006.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Jin Tian and Judea Pearl. A general identification condition for causal effects . eScholarship, University of California, 2002.

Zhi Wang, Chicheng Zhang, and Kamalika Chaudhuri. Thompson sampling for robust transfer in multi-task bandits. arXiv preprint arXiv:2206.08556 , 2022.

Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In International Conference on Artificial Intelligence and Statistics , pages 1531-1539. PMLR, 2021.

Kan Xu and Hamsa Bastani. Learning across bandits in high dimension via robust statistics. arXiv preprint arXiv:2112.14233 , 2021.

Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandit: a causal approach. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , pages 1778-1780, 2017.

Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. arXiv preprint arXiv:2110.05690 , 2021.

Appendix for "Transportability for Bandits with Data from Different Environments"

This Appendix includes

Related work in Appendix A.

Discussion on limitations and analysis of misspecification in Appendix B.

Proofs in Appendix C.

Details on the data generating mechanisms Appendix D.

Details on the Gibbs sampler in Appendix E.

A Related work

In the bandit literature, problems of transfer learning in which existing experience and knowledge is used to improve the performance of an agent appear under the rubric of multi-task learning and meta-learning. Typically, in the context of multi-task learning the agent aims to solve a prescribed set of related bandit tasks with shared structure. For example, [41] consider the setting in which multiple players interact in an environment with the property that each player has slightly different associated reward distributions. [40] extend this approach to the Thompson sampling algorithm with an assumption that reward distribution between players are close but not identical. Similar ideas can be found in the contextual case in which, e.g. [13] propose to leverage similarities in contexts for different arms and improve prediction of reward distributions from contexts. In the context of meta-learning, the agent is designed to work well on an arbitrary number of tasks from a common environment ( i.e. sampled from a prescribed distribution), relying on already completed tasks from the same environment. For example, [4, 20, 33] assume a hierarchical bandit structure in which parameters governing multiple bandit instances are sampled i.i.d. from a meta-prior. The authors demonstrate that updates on the meta-prior from one instance can benefit other instances and that regret bounds over a sequence of instances can be established. [30] adopt a similar setting to infer a Lipschitz continuity constant which can be used to derive scale free regret bounds. Others, e.g. [10, 42], have assumed that structural parameters of each instance can be decomposed into a shared component and an instance-specific component, and learned with high-dimensional regularization schemes.

Existing methods adopt a relatively wide range of assumptions on the 'relatedness' of different bandit instances or players that include, as surveyed above, the existence of a meta-distribution across bandit instances, assumptions on the pairwise similarities in the reward distributions of different players, assumptions on the decomposition of bandit parameters, and assumptions on the distribution of contextual variables and their association with rewards. The environment and causal structure typically remain invariant across bandit instances, and only small and specific set of changes in distribution are allowed across bandit instances.

The proposed approach aims to tackle a more general setting in which we allow for arbitrary changes in the causal mechanisms underlying each environment as long as their location can be established and encoded in selection diagrams, i.e. graphs that describe the difference in structure across domains without constraining their form. In the causality literature, these analyses fall under the transportability umbrella. Several authors have demonstrated the power of this approach to identify causal effects, predict counterfactual distributions and, more generally, make inference across different environments [3, 11, 32, 5]. In the bandit literature, there is also an extensive literature on various ways of exploiting knowledge of a causal graph for a single environment, known as causal bandits [23, 36, 28, 27, 24, 26, 12, 29, 7, 18]. Within this literature the causal graph specifies a dependency structure between different actions that allows an agent to identify redundant actions and ultimately more efficient exploration, improving regret bounds by a multiplicative factor. Practical examples of the use of causal graphs (within decision-making problems) are prevalent in the contexts of clinical trials, healthcare, and advertising. We note, in particular, the examples in Figs. 2, 3, 4, in [22] that encode the design of case-cohort studies and clinical trials used in the MORGAM study [14] using causal graphs for the estimation of causal effects. Separately, [21] consider the

International Stroke Trial and the known causal associations between relevant features for policy optimization. In the context of advertising, [8] describes the use of causal methods with several detailed examples. Specifically, Figs. 3, 4, and 6 in [8] show examples of causal graphs that may be defined for particular computational advertising applications. See also [5] for a bandit algorithm leveraging the computational advertising graphs in [8], and[36, Sec. 5.2] for similar causal treatments also in the context of advertising. So far, however, all bandit algorithms exploiting causal knowledge have been studied in the 'single environment' setting without access to prior data. Variations among existing proposals are due to the class of graphs or the extent of knowledge of the graph, e.g. with or without knowledge of the full causal graph, with or without unobserved confounding, etc. One may interpret the proposed approach as extending the causal bandit formalism to problems in which data from multiple different environments is available prior to an agent experimenting in a target environment, all of which are, as in the causal bandit literature, described by underlying causal models.

(a)

P

˚

p

z

q

arbitrary.

(b)

P

˚

p

z

q P r

0

.

2

,

0

.

8

s

.

(c)

P

˚

p

z

q P r

0

.

3

,

0

.

7

s

.

(d)

P

˚

p

z

q P r

0

.

4

,

0

.

6

s

.

50

60

70

60

70

40

50

50

60

30

40

40

50

20

30

30

30

40

20

20

20

10

10

10

10

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

0

0.0

0.2

0.4

0.6

0.8

1.0

Expected reward under do(x=1)

Expected reward under do(x=1)

Expected reward under do(x = 1)

Expected reward under do(x=1)

B Limitations and further analysis

Knowledge of structure may detract from the appeal of active experimentation and might limit the extent to which the proposed approach can be applied. Although in practice several techniques can be used to alleviate this problem such as causal discovery or considering multiple causal graphs as potential causal explanations for the phenomenon of interest, significant domain knowledge will in general be necessary for consistently using prior data. This observation holds for causal bandits more generally.

B.1 Misspecification

Some degree of misspecification, however, can be handled without voiding guarantees on performance improvements, especially when the assumed causal graphs and selection diagrams form super-models for the underlying phenomenon of interest.

Starting with causal graphs: a causal graph ˜ G a,b defined as a graph p V , ˜ E q is said to be a super-model of a causal graph G ' p V , E q E Ď ˜ E . For any causal graph G it holds that adding an edge, creating a new graph ˜ G , creates a larger set of SCMs M p ˜ G q that subsumes M p G q , i.e. M p ˜ G q Ă M p ˜ G q , and as a consequence any SCM with non-zero mass under P M p G q also has non-zero mass under P M p ˜ G q . This implies that prior distributions P M ' M p ˜ G q p E P M r Y x s | ¯ v q remain valid whenever the causal graph ˜ G is a super-model of the true underlying graph G as the space of models defined by G is included in the space of models defined ˜ G . For example, consider a scenario in which source and target domains are equal with an underlying causal graph given by G : ' t X Ñ Y u . Prior to experimentation with knowledge of the graph and source data distribution P p x, y q , reward probabilities under agent intervention can be uniquely computed, i.e. P p y | do p x qq ' P p y | x q . However, if we are unsure about the presence of an unobserved confounder between X and Y , we may instead consider a supermodel G : ' t X Ñ Y, X glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] Y u under which reward probabilities under agent intervention can be determined to be distributed in the interval r P p x, y q , P p x, y q ` 1 ´ P p x qs , that, in particular, includes the true value P p y | do p x qq ' P p y | x q . Inference based on ˜ G instead of G leads to less informative reward probabilities but remains correct and informative with respect to methods not leveraging prior data as 0 ď P p x, y q ď P p y | do p x qq ď P p x, y q ` 1 ´ P p x q ď 1 .

A similar reasoning can be applied to selection diagrams and selection nodes, as the latter only indicates a potential discrepancy across domains. A selection diagram ˜ G a,b defined as a graph p V Y ˜ S , ˜ E q is said to be a super-model of a selection diagram G a,b ' p V Y S , E q if S Ď ˜ S and E Ď ˜ E . In words, super-models can have more edges or selection nodes. If a researcher is unsure of whether a causal mechanism differs or not, they may still conduct correct

X

Y

Z

S

Z

Figure 7:

G

a,

˚

inference by being conservative and assuming that a discrepancy exists. Under these types of misspecification, where assumed selection diagrams represent super-models of the underlying data generating process all claims and propositions hold and we can guarantee to never under-perform methods that do not consider prior data or their structure. For example, consider a system with binary variables X,Y,Z P t 0 , 1 u whose causal association and differences across domains are given by Fig. 7. In reality, target and source environments coincide and, with access to batch data from the source environment π a , one could uniquely identify expected rewards by computing E P ˚ Y x ' ř y y ř z P a p y | x, z q P a p z q . However, we suspect some degree of discrepancy in the

X

Z

Y

S

X

(a)

(b)

(c)

20.0

Run time (seconds)

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

2

35102050100

Number of actions

8

Run time (seconds)

７６５４３２１

0

500

10002000

500010000

causal assignment of Z across environments, and will consider inference of expected rewards under various degrees of discrepancy, e.g. one may explicitly assume that either P ˚ p z q is arbitrary or for example assume P ˚ p z q P r a, b s . We illustrate various such scenarios in Fig. 6. Each panel, from right to left increases allows for a greater level of uncertainty that results in a correspondingly wider distribution of expected reward prior to experimentation. The corresponding regret of bandit algorithms that incorporate these bounds on Z in P ˚ to warm start the distribution of expected regret prior to experimentation are given in Fig. 8. We observe that the wider the prior the higher the corresponding regret, which illustrates how degrees of prior knowledge influence regret.

Using the same reasoning, it holds that if instead the researcher misses edges or incorrectly assigns priors on the causal mechanisms to be expected in the target domain, prior inference will be incorrect in general. For example, if distributions of expected reward in Fig. 6 do not cover the true expected reward, then a bandit algorithm will need to correct for such a biased prior and we can expect cumulative rewards to be lower than those of an algorithm that ignores prior data and structure. The extent to which the corresponding regret of a bandit algorithm incorrectly making assumptions on structure and discrepancies across environments will depend on the underlying causal graph and functional associations between variables.

10

p*(z) ∈[0,1]

8

P* (z) E [0.2, 0.8]

Regret

6

P*(z) E [0.3, 0.7]

P*(z) E[0.4,0.6]

4

2

0

0

25

50

75 100 125 150 175 200

Rounds

Most of the discussion in the main body of this paper has assumed knowledge of a discrepancy between environments without considering its magnitude, although the previous example shows that knowledge of bounds or distributions of some of the variables that differ in the target domain can be incorporated. It is worth noting that other kinds of domain knowledge may be available, such as in the functional form of an association in the target domain, specific magnitudes of differences expressed within such a functional form, etc. It would be an interesting task to consider how to use that information optimally.

B.2 Run time evaluations

Reward distributions P ˚ p y x q induced by any underlying SCM can be expressed by a corresponding discrete SCM in which P p U q is discrete and F is a discrete mapping between finite spaces. The complexity of the corresponding parameterization depends on the underlying selection diagram and dimensionality of observed variables.

In particular, in a given iteration of the Gibbs sampler, posterior updates are done for each parameter separately so that computational time is proportional to the parameter count, approximately, which in turn is determined by the cardinality of variables as well as the structure of the graph. For a fixed graph, assuming that each update requires a small constant amount of time to compute, we could therefore establish analytically how computational time scales with the cardinality of variables. For arbitrary graphs, an analytical result is in general more involved as the parameter count increases differently depending on the local structure of each variable. As an example for illustration, consider the graph in Fig. 9a where X is an action variable, Z is a contextual variable, and Y is a reward variable. Following the parameterization in Corol. 1, the cardinality of parameters is defined as follows: | θ U | ' | Ω X | ¨ | Ω Z | ¨ | Ω Y | , | ξ X | ' | Ω X | ¨ | Ω Z | ¨ | Ω Y | , | ξ Z | ' | Ω X | , | ξ Y | ' | Ω X | ¨ | Ω Z | ¨ | Ω Y | ¨ | Ω Z | .

We would expect run time to increase linearly with the cardinality of variables X,Y and to increase 'slower than quadratically' with the cardinality of Z . Fig. 9b gives run time evaluations of the proposed bandit algorithm using 1000 rounds of experimentation as a function of the number of actions. Fig. 9c evaluates the influence of the size of prior data on run time by plotting the time required for drawing 1000 samples from the prior P p¨ | ¯ v q with increasing prior data size (in addition to 1000 samples Gibbs samples that are discarded as burn-in). For both Figs. 9b and 9c we observe run time to scale approximately linearly with the cardinality of the action space and sample size.

C Theoretical results

C.1 Preliminaries

Counterfactuals and optimal actions We have defined in Sec. 1.1 the fact that intervening in an SCM results in a new model that differs in the causal mechanisms subject to interventions. In this section, we extend this notion to consider models derived from interventions that replace a causal mechanism with another function, not necessarily a constant. This gives consistent definitions for counterfactual random variables of the form Y X that appear in Sec. 3.

Definition 4 (Intervened model) . Let M be an SCM, ˆ U Ď U , X P V , and ˆ X : ˆ U Ñ Ω X a function. Then, M ˆ X , called the intervened model of M subject to ˆ X , is identical to M , except that the function f X is replaced with the function ˆ X .

Uncertainty in M ˆ X is similarly encoded by the distribution P p U q which when averaged over leads to random variables Y ˆ X for Y P V that describes the random variable Y under a model M ˆ X . This definition is tightly related to that of potential outcome as define in Sec. 1.1, but the former explicitly allows for interventions that do not necessarily fix the variable X to a constant value. Probabilities in this model can be computed using the chain rule

And similarly expectations are given by

In turn, which action is optimal is represented by a random variable ˜ X : Ω ξ ˆ Ω θ ÞÑ Ω X where Ω ξ ˆ Ω θ defines the space of possible models M : ' M p ξ , θ q P M p G π ˚ q that parameterize the deployment environment π ˚ . For example, P M ' M p G π ˚ q p ˜ X ' x q ' P M ' M p G π ˚ q p E P M r Y x s ą E P M r Y x 1 s , @ x 1 P Ω X zt x uq where P M ' M p G π ˚ q is a probability mass function defined over the space of possible models M p G π ˚ q .

Information Theory Definitions This section provides a definition of all the information-theoretic terms used throughout and shows that they can be generalized to mixed quantities that include discrete and continuous random variables in a consistent manner. For a discrete random variable X P Ω X , the Shannon entropy and its conditional counter parts are defined as

With a slight abuse of notation, we will define the filtered entropy as

where ¯ v is a dataset.

Given two probability measures P and Q , where P is absolutely continuous w.r.t. Q so that the Radon-Nikodym derivative dP dQ is well defined, the Kullback-Leibler divergence is

which allows the standard definition of mutual information :

where the subscript P that denotes the probability distribution used is sometimes omitted if unambiguous. Equivalently, we can define the mutual information and conditional mutual information by

and we will also need the filtered mutual information ,

where we will typically evaluate I P p¨| ¯ V t , ¯ v q p X ; Y q : ' D p P p X,Y | ¯ V t , ¯ v q} P p X | ¯ V t , ¯ v q P p Y | ¯ V t , ¯ v qq where recall ¯ V t : ' t V X p 1 q , . . . , V X p t ´ 1 q u denotes the random variables associated with the agent's history of interactions with π ˚ up to round t . The conditional mutual information is related to the filtered mutual information by an expectation: E r I P p¨| Z q p X ; Y qs ' I p X ; Y | Z q , with respect to P p Z q . We will frequently use this fact to show that

where the expectation is taken with respect to P p ¯ V t | ¯ v q . Perhaps the most important property of mutual information will be the chain rule,

We will need to evaluate I p θ , ξ ; Z q , where Z is a discrete random variable, e.g. ¯ v t or V p t q , θ is continuous, and ξ is discrete. We can evaluate this mutual information by writing

which are all well defined.

C.2 Proofs

For readability, we restate all propositions.

Proposition 1 (restated). Let R T denote the regret incurred by following Thompson sampling (Alg. 1). For any T P N and Γ ě Γ t , then

where H p ˜ X | ¯ v q is the conditional entropy of ˜ X given ¯ v .

Proof. This analysis extends [35, Prop. 1] to account for the case in which prior data is conditioned upon. We first bound the expected regret by using the law of total expectations and introducing the mutual information between the optimal action and the action observation tuple,

By using the KL divergence definition of mutual information and using the fact that Thompson sampling precisely chooses actions according the their probability of being optimal, it can be shown that,

where P p X ' x q ' P p ˜ X ' x q by definition of the policy used by Thompson sampling. Let the information ratio Γ t is defined as an upperbound on the ratio

By writing Γ ě Γ t , @ t , applying this bound yields

where the second to last inequality follows from the Cauchy-Schwartz inequality and the last inequality follows from the fact that,

by the chain rule for the mutual information and given the non-negativity of the entropy.

Proposition 2 (restated). Let R T denote the regret incurred following the policy defined by Alg. 1. For any T P N , if Eq. (7) holds with Γ ě Γ t for all t ,

Proof. This proof uses a different characterization of the per round regret to explicitly consider the influence of model parameters p θ , ξ q on information gain. Assume instead that there exists Γ t such that

where ϵ t ą 0 is an additional slack term. The proof strategy follows that of [25, Prop. 2] where unconditional regret bounds were shown. Given Eq. (22) the following derivation holds, by definition by the law of iterated expectations by definition of Γ t and ϵ t by Cauchy-Schwarz's inequality by definition of the expectation of I P ff by the chain rule for mutual information .

Proposition 3 (restated). Fix δ ą 0 and choose Γ t such that ˇ ˇ Y x ´ E r Y x | ¯ V t , ¯ v s ˇ ˇ ď Γ t 2 b I P p¨| ¯ V t , ¯ v q p θ , ξ ; Y x q for all x P Ω X simultaneously with probability greater than 1 ´ δ . Then Alg. 1 chooses actions X p t q that satisfy

where B ě 0 is such that sup y,y 1 P Ω Y y ´ y 1 ď B .

Proof. This proposition extends [25, Lem. 3] to account for the case in which prior data is conditioned upon. Thompson sampling, by definition, samples an action according to its probability of being optimal with the current distribution of parameter values,

where ˆ M : ' M p ˆ ξ , ˆ θ q defines the SCM that is obtained with the current sample ˆ ξ , ˆ θ ' P p ξ , θ | ¯ v t , ¯ v q from the posterior distribution at round t , while M π ˚ refers to the true underlying SCM of the environment. Expectations with respect to the distributions P ˆ M and P π ˚ are conditioned on a specific model of the environment. Define,

where Y x,M p ξ , θ q refers to the random variable Y in SCM M p ξ , θ q in which we intervene to set X to x , and Y x,M refers to the random variable Y in SCM M in which we set X to x .

The probability matching property of Thompson sampling implies that P p ˆ ξ , ˆ θ P E | ¯ v t , ¯ v q ě 1 ´ δ { 2 . It follows then, using the same proof strategy as [25, Lemma 3], that

where the last equalities follow from the conditional independence between X p t q and p θ , ξ q conditioned on ¯ V t , ¯ v .

This proposition might guide us to obtain a per-period regret bound that in particular determines a choice for the noise term ϵ t , which results in an additive term inProp. 2. A common choice for ϵ t is 1 { T where T is the number of experimentation rounds and therefore the additive term in the regret is simply a constant. This is also the choice made by [25]. In this way, the regret bound is tuned with knowledge of the horizon, which is common in online learning.

C.3 Information ratio computation for specific example

The theoretical statements presented in the main body of this work only implicitly relate bounds on regret with the assumed environment discrepancies assumed in selection diagrams. It is challenging in general to provide more explicit connections. Although similarity in structure between domains is explicitly encoded in selection diagrams, the association between discrepancies and posterior distributions on parameters or reward probabilities learned from prior data for an arbitrary input selection diagram are complex. For instance, it doesn't necessarily hold that few discrepancies lead to narrow reward probability posterior distributions, and conversely it doesn't necessarily hold that large discrepancies, e.g. many selection nodes, lead to wide reward probability posterior distributions.

In particular, consider an extreme example in which source and target domains are equal and described by G : ' t X Ñ Y, X glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] Y u . Even with infinite data and no discrepancy, it can be shown that posterior reward probabilities P ˚ p y | do p x qq are bounded in the interval r P p x, y q , P p x, y q` 1 ´ P p x qs which might be arbitrarily wide depending on the underlying SCM. In contrast, in other examples with certain graphs in which source and target domains differ on an arbitrarily large set of causal mechanisms (that are, however, irrelevant for the computation of posterior reward probabilities) posterior reward probabilities could be made arbitrarily narrow with sufficient data.

The information ratio Γ can highlight the trade-off between regret and graph structure. For specific graphs, a bound on the information ratio Γ can be analytically computed and used to convey some of the performance gains to be expected, for intuition. Recall that the information ratio is defined as a scalar Γ t such that,

The information ratio can be bounded: 1 { 2 ď Γ t ď | Ω X |{ 2 , and describes how much information is revealed about reward distributions when an arm is pulled: on one end Γ t ' 1 { 2 in problems with full information arise when the outcome Y x is perfectly revealed by observing Y x 1 , and on the other end Γ t ' | Ω X |{ 2 in problems where random variables Y x and Y x 1 are independent for any

x ‰ x 1 . Bounds on the information ratio have also been shown for linear bandits [35]. With our parameterization of P π ˚ p y x q the probability of exogeneous variables encoded in θ is shared while ξ is not, thus some information about reward distributions is expected to be gained on all arms in each MABround, and 1 { 2 ă Γ t ă | Ω X |{ 2 .

In the following we will quantify the amount of sharing between reward distributions and compute a corresponding a smaller upperbound for the information ratio for a specific graph in which the computation is tractable following the proof strategy of [35, Proposition 5]. This illustrates more precisely the regret gains that can be expected from having a shared parameterization for reward distributions.

X

Y

Z

S

Z

We consider the selection diagram with domain-specific confounding given in Fig. 10 and the computation of P ˚ p y | do p x qq ' ř z P Ω Z P a p y | x, z q P ˚ p z q given knowledge of P a p x, y, z q . This graph is interesting because we may directly parameterize the conditional distributions involved in P ˚ p y | do p x qq instead of considering its underlying SCM. Let Y be binary for simplicity (all arguments hold also more generally). For the purposes of this discussion, we will assume that P a p y | x, z q can be approximated arbitrarily well from sufficient prior data. It holds that

where ϕ p x q P R | Ω Z | is a column vector with i -th entry P π a p Y ' 1 | x, z ' i q and p P R | Ω Z | is a column vector with i -th entry P p Z ' i q . The latter has some distribution in the interval r 0 , 1 s and in particular we denote its mean by µ : ' E r p s and its mean conditioned on the optimal action being x j by µ j : ' E r p | ˜ X ' x j s . Recall that ˜ X denotes the optimal action.

Let α i : ' P p ˜ X ' x i q and define M P R | Ω X |ˆ| Ω X | by its p i, j q -th entry,

All expectations are taken with respect to P p¨ | ¯ V t , ¯ v q . M can therefore be written as a product of two matrices of rank | Ω X | . With this definition, it was shown by [35, Proposition 5] that,

leading to the fact that the information ratio,

Now, | Ω Z | ăă | Ω X | in certain applications which shows that the information ratio may be smaller than the worst-case value of | Ω X |{ 2 due to sharing of P ˚ p z q across different actions x .

and that,

D Details on the data generating mechanisms

Bow graph with domain-specific confounding. To generate source data we choose an SCM M a compatible with the graph specified as follows: P p u Z q , P p u X , P p u Y , P p u XY q are given by independent Gaussian distributions with mean 0 and variance 1, and each observation p z, x, y q is generated from p u Z , u X , u Y , u XY q using the structural assignments: z Ð ✶ t u Z ă 1 u , x Ð t z ` u XY ´ u X ą 0 u , y Ð t x ´ 0 . 5 z ` 2 u XY ą 0 u .

✶ Hypertension example. To generate data from the patient population π a we choose an SCM M a compatible with the graph specified as follows: P p u Z q , P p u X , P p u W , P p u XY q are given by independent Gaussian distributions with mean 0 and variance 1, and each observation p z, x, w, y q is generated from p u Z q , u X , u Y , u XY q using the structural assignments: z Ð ✶ t u Z ą 0 u , x Ð int t 0 . 5 z ` ✶ t u XY ą 0 u ` 2 ✶ t u XY ą 0 u ´ ✶ t u X ą 0 u ą 0 u ` 2 ✶ t u X ą 1 u ` 1 u , w Ð ✶ t 0 . 3 x ´ u W ´ 0 . 8 ą 0 u , y Ð ✶ t w ´ 0 . 5 z ` u XY ą 0 u . int t¨u stands for the integer part of the content of the brackets.

✶ ✶ The deployment domain π ˚ is given by a different SCM M ˚ that varies in the causal mechanism relating to Z . It is given by the following SCM: P p u Z q , P p u X , P p u Y , P p u XY q are given by independent Gaussian distributions with mean 0 and variance 1, and each observation p z, x, y q is generated from p u Z q , u X , u Y , u XY q using the structural assignments: z Ð ✶ t u Z ă 0 u , x Ð ✶ t z ` u XY ´ u X ą 0 u , y Ð t x ´ 0 . 5 z ` 2 u XY ą 0 u .

The domain π ˚ in which the MAB is deployed is given by a different SCM M ˚ that varies in the causal mechanism relating to Z and W with respect to M b , and varies in W with respect to M a . It is defined by: P p u Z q , P p u X , P p u W , P p u XY q are given by independent Gaussian distributions with mean 0 and variance 1, and each observation p z, x, w, y q is generated from p u Z q , u X , u Y , u XY q using the structural assignments: z Ð ✶ t u Z ą 0 u , x Ð int t 0 . 5 z ` ✶ t u XY ą 0 u ` 2 ✶ t u XY ą 0 u´ t u X ą 0 u` 2 t u X ą 1 u` 1 u , w Ð t 0 . 2 x ´ u W ´ 1 . 8 ą 0 u , y Ð t w ´ 0 . 5 z ` u XY ą 0 u .

✶ The domain π ˚ in which the MAB is deployed is given by a different SCM M ˚ that varies in the causal mechanism relating to A with respect to M a . It is defined by: P p u Z q , P p u X , P p u W , P p u XW , P p u WY , P p u A q are given by independent Gaussian distributions with mean 0 and variance 1, and each observation p z, x, w, a, y q is generated from a sample u Z , u X , u W , u XW , u WY , u A using the structural assignments: w Ð ✶ t u XW ` u WY ą 0 u , z Ð ✶ t u Z ` w ą 0 u , a Ð ✶ t u A ą 0 . 5 u , x Ð int t 0 . 5 z ´ 0 . 5 a ` ✶ t u XW ą 0 u ` 2 ✶ t u XW ą 0 . 5 u ´ ✶ t u X ą 0 u ` 1 u , y Ð ✶ t 0 . 2 x ´ 0 . 5 a ` u WY ´ 1 ą 0 u .

✶ ✶ ✶ ✶ Digital advertising example. This example considers data from two domains. Data from the source domain π a is given by a SCM M a compatible with the graph that is specified as follows: P p u Z q , P p u X , P p u W , P p u XW , P p u WY , P p u A q are given by independent Gaussian distributions with mean 0 and variance 1, and each observation p z, x, w, a, y q is generated from a sample u Z , u X , u W , u XW , u WY , u A using the structural assignments: w Ð ✶ t u XW ` u WY ą 0 u , z Ð ✶ t u Z ` w ą 0 u , a Ð ✶ t u A ą 0 u , x Ð int t 0 . 5 z ´ 0 . 5 a ` ✶ t u XW ą 0 u` 2 ✶ t u XW ą 0 . 5 u´ ✶ t u X ą 0 u ` 1 u , y Ð t 0 . 2 x ´ 0 . 5 a ` u WY ´ 1 ą 0 u .

E Gibbs sampling

This section gives the derivation of all conditionals using our parameterization of causal effects for the bow graph with confounding example.

The query of interest is given by P ˚ p y x ' 1 q which is first approximated using source data ¯ v and then updated using interventional data v x p 1 q , v x p 2 q , . . . collected by the MAB in the deployment environment π ˚ .

Its parameterization, following Corol. 1, is given by

where ξ p x,z,u xy q Y and θ u xy parameters are shared across source and deployment environments, while ✶ t ξ p u z q Z ' z u and θ u z is specific to the deployment environment. We start by approximating the posterior of all relevant parameters, that is ξ p x,z,u xy q Y and θ u xy , given ¯ v , before interacting with the deployment environment π ˚ . Note that prior data ¯ v is not relevant for estimating ✶ t ξ p u z q Z ' z u and θ u z and thus we maintain uniform prior distributions over these parameters at this stage. In the following, we give the derivation of the complete conditionals over ξ p x,z,u xy q Y and θ u xy .

Sampling from P p ¯ u xy , ¯ u z | ¯ v , ξ , θ q . Let ¯ u xy ' t u p n q xy , n ' 1 , . . . , N u and ¯ u z ' t u p n q z , n ' 1 , . . . , N u denote N independent samples of U xy and U z respectively, one corresponding to each observation v p n q ' p x p n q , y p n q , z p n q q , where is the number of prior data samples. The complete conditional can be derived following the functional dependencies in the underlying SCM given by the causal graph,

✶ ✶ ✶ where we have replaced the probabilities with the corresponding parameters that are used to define them.

Sampling from P p ξ p x,z,u xy q Y | ¯ v , ¯ u , θ q . Similarly, for fixed x, z, u xy , parameter ξ p x,z,u xy q Y is mutually independent of any other parameter in ξ given ¯ v , ¯ u , θ and can be sampled separately. Recall that by definition of the underlying SCM ξ p x,z,u xy q Y represent a deterministic mapping between inputs x, z, u xy and output y P Ω Y . The value ξ p x,z,u xy q Y P Ω Y is therefore implicitly determined by the current values ¯ v , ¯ u : if there exists a tuple p x p n q ' x, z p n q ' z, u p n q xy ' u xy , y p n q ' y q for some n ' 1 . . . , N , then by definition ξ p x,z,u xy q Y : ' y with probability 1. If no such tuple exist, then the distribution of ξ p x,z,u xy q Y remains uniform over its domain Ω Y as none of the data points carries information as to the mapping p x, z, u xy q ÞÑ y .

Sampling from P p θ u xy | ¯ v , ¯ u , θ q . The conditional distribution over θ u xy given ¯ v , ¯ u is given by a Dirichlet distribution following the conjugacy of it with regard to categorical distributions of U xy and can be updated by adding the counts of each outcome ¯ u xy of U xy in the current sample to the corresponding prior concentration parameters of the Dirichlet prior distribution.

This process eventually forms a chain of samples from the correct posterior distribution of each parameter. At this stage, the MAB is deployed in π ˚ and all parameters may be updated both with prior data ¯ v as well as with additional samples v x collected online in every round of experimentation. Offline and online data points are different in kind, and contribute to updating parameters differently. For shared parameters, both types of data may be used while for parameters specific to π ˚ only newly collected interventional data samples will be relevant in posterior computations.

P p ξ p u z q Z | v x p 1 q , . . . , v x p t ´ 1 q , u x p 1 q , . . . , u x p t ´ 1 q q using the same intuition as described above. Similarly, θ u xy is updated using the posterior P p θ u xy | ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q while θ u z is updated using the posterior P p θ u z | ¯ u , u x p 1 q , . . . , u x p t ´ 1 q q ' P p θ u z | u x p 1 q , . . . , u x p t ´ 1 q q .

Peer reviews:
comments: SUMMARY:
The paper analyses how bandits can exploit causal similarities (need not be fully known functionally) across different environments to improve their regret bounds. In particular, previously collected data from related environments can improve learning.

STRENGTHS:
Strengths
1. Use of domain discrepancy and selection diagrams to formally treat the different environments.
2. The proposed tTS conforms nicely with how a bandit can use priors and recently tried arms to inform its posteriors.

WEAKNESSES:
None

QUESTIONS:
None

LIMITATIONS:
None identified.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-26

comments: SUMMARY:
A framework is presented so that bandits can use data from different environments, by exploiting the causal relationship between those environments.

STRENGTHS:
The contribution and problem statement are clear.

WEAKNESSES:
Experiments seem to be conducted on toy data only. How useful is the method in practice, on real data?

QUESTIONS:
cf above

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of leveraging data from many different environments to warm start a bandit algorithm. The paper assumes that the environments share the same variables, and the relationships between variables in related environments can be captured by structural causal models (SCMs). Taking a Bayesian perspective, the paper defines a probability distribution over unknown quantities in the target SCM and constrains the probability distribution using the information obtained from the related environments. This probability distribution is then used to warm-start a Thompson sampling algorithm.\n\nSTRENGTHS:\n1. This paper has interesting theoretical contributions. The authors provide an algorithm that leverages data from related environments and are able to prove a sub-linear regret bound and the regret bound depends on a term that captures how informative the related environments are for the target environment.\n\n2. Compelling experimental results.\n\n3. The paper is overall clear and well-written.\n\nWEAKNESSES:\n1. This work relies on having quite detailed information from the related environments– the SCM and the selection diagrams for each environment. Having this type of information is somewhat unrealistic. Nevertheless, the contribution of the paper is conceptual/theoretical and may pave the way for future work that requires less detailed information.\n\n2. Furthermore, it is also somewhat unrealistic that any of the relationships between the variables in the related environment are exactly the same as the relationship as relationships in the target environment. Would it be possible to place looser restrictions from the prior data than the equality constraints in Eq 2?\n\nQUESTIONS:\n1. In the motivating example in lines 42-60, it would be helpful if the authors could add a comment that this problem arises due to the fact that the clinical trial population and historical data population differ across observables (or unobservables).\n\n2. [Nit] $S_{Z}$ is not defined in the text until Section 2 but appears in Figure 1, which is referenced in Section 1.\n\n3. Would it be possible for the authors to contextualize their perspective on distribution shift within the broader literature on generalizability/transportability (e.g., Stuart, et. al. 2011, Tipton, et. al, 2013, Tipton et. al., 2014)? For example, can the different environments vary across unobservable attributes (confounders), or can they only differ in observable attributes?\n\n\nReferences:\n\nStuart, Elizabeth A., et al. \"The use of propensity scores to assess the generalizability of results from randomized trials.\" Journal of the Royal Statistical Society: Series A (Statistics in Society) 174.2 (2011): 369-386.\n\nTipton, Elizabeth. \"Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts.\" Journal of Educational and Behavioral Statistics 38.3 (2013): 239-266.\n\nTipton, Elizabeth. \"How generalizable is your experiment? An index for comparing experimental samples and populations.\" Journal of Educational and Behavioral Statistics 39.6 (2014): 478-501.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper considers the online bandit problem with 
additional batch/observational data, where 
the additional data could be from different (but related) 
environments. The authors 
present a representation of the interventional distribution, based on which one can sample from the posterior distribution 
of the SCMs and select an action based on the realized model. 
It is shown that the resulting regret is sublinear, and 
the improvement upon an algorithm without using prior data is explicitly 
dependent on the "relevance" of the other environments.

STRENGTHS:
1. This paper considers an interesting problem: how to leverage observational 
data to improve the online bandit algorithm. 
2. The proposed method makes use of prior data in a clean way, and the 
theoretical result shows an explicit dependence on the relevance of 
the environments from which the prior datasets are generated, while 
the algorithm itself is agnostic to this knowledge.

WEAKNESSES:
1. This paper is rather notation-heavy, and it is a bit hard 
for readers not familiar with the language used therein.
2. The proposed method introduces an additional computational burden.

QUESTIONS:
1. I was wondering how the computation time scales with the cardinality of different variables. Appendix B.2 has briefly mentioned this --- I am curious if it would be possible to have an analytical result.
2. It might also be helpful to compare the computational time of different methods 
in the simulations (in addition to the ones presented in Appendix B.2).

Minor: 
Is there a typo in Algorithm 1? "$\mathcal{G} ** a$"

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07