Title: Approximate Allocation Matching for Structural Causal Bandits with Unobserved Confounders
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

causal Inference

sequential decision-making
Abstract: Structural causal bandit provides a framework for online decision-making problems when causal information is available. It models the stochastic environment with a structural causal model (SCM) that governs the causal relations between random variables. In each round, an agent applies an intervention (or no intervention) by setting certain variables to some constants and receives a stochastic reward from a non-manipulable variable. Though the causal structure is given, the observational and interventional distributions of these random variables are unknown beforehand, and they can only be learned through interactions with the environment. Therefore, to maximize the expected cumulative reward, it is critical to balance the explore-versus-exploit tradeoff. We assume each random variable takes a finite number of distinct values, and consider a semi-Markovian setting, where random variables are affected by unobserved confounders. Using the canonical SCM formulation to discretize the domains of unobserved variables, we efficiently integrate samples to reduce model uncertainty. This gives the decision maker a natural advantage over those in a classical multi-armed bandit setup. We provide a logarithmic asymptotic regret lower bound for the structural causal bandit problem. Inspired by the lower bound, we design an algorithm that can utilize the causal structure to accelerate the learning process and take informative and rewarding interventions. We establish that our algorithm achieves a logarithmic regret and demonstrate that it outperforms the existing methods via simulations.

Approximate Allocation Matching for Structural Causal Bandits with Unobserved Confounders

Abstract

Structural causal bandit provides a framework for decision-making problems when causal information is available. It models the stochastic environment with a structural causal model (SCM) that governs the causal relations between random variables. In each round, an agent applies an intervention (or no intervention) by setting certain variables to some constants, and receives a stochastic reward from a non-manipulable variable. Though the causal structure is given, the observational and interventional distributions of these random variables are unknown beforehand, and they can only be learned through interactions with the environment. Therefore, to maximize the expected cumulative reward, it is critical to balance the explorationversus-exploitation tradeoff. We consider discrete random variables with a finite domain and a semi-Markovian setting, where random variables are affected by unobserved confounders. Using the canonical SCM formulation to discretize the domains of unobserved variables, we efficiently integrate samples to reduce model uncertainty, gaining an advantage over those in a classical multi-armed bandit setup. We provide a logarithmic asymptotic regret lower bound for the structural causal bandit problem. Inspired by the lower bound, we design an algorithm that can utilize the causal structure to accelerate the learning process and take informative and rewarding interventions. We establish that our algorithm achieves a logarithmic regret and demonstrate that it outperforms the existing methods via simulations.

1 Introduction

Sequential decision-making in uncertain environments is one of the most fundamental problems across scientific disciplines, including robotics, economics, social science, clinical trials, and agriculture [32, 26, 36, 7, 3]. In almost all these applications, it is required to implement interventions to control some aspects of the system to optimize an outcome of interest. In economics and political science, the government could use fiscal policy including changing the interest rate and taxation to achieve regulation and control of a country's economy. In this paper, we deal with designing an adaptive policy in an uncertain stochastic environment to learn and apply optimal interventions.

In sequential decision-making problems, it is critical to balance the explore-versus-exploit tradeoff, which deals with choosing between informative options and empirically more rewarding alternative options. The multi-armed bandit (MAB) [33, 30, 18] is a classical problem formulation for achieving such a tradeoff. In MAB, a decision-maker sequentially chooses an action corresponding to pulling

an arm from a set of arms, each generating stochastic rewards with probability distribution unknown to the decision-maker. The objective is to achieve the maximum cumulative reward over a time horizon, or equivalently to minimize the regret, which measures suboptimality in cumulative rewards against selecting the best action all the time. The classic stochastic bandit formulation only considers the connection between an action and its expected reward. However, in real-world applications, an action can affect the entire system in a more complex way. As the agricultural example given in [19], changing the moisture level could affect the temperature and nutrient absorption, which are also crucial to crop yields. The relations between such factors cannot be characterized by the MAB.

Causal knowledge enables us to model the causal relations between multiple factors in the environment. In machine learning research, it has been noted that causal models could help improve the generalization performance when data distribution shifts [11], and address unstable translation of successful reinforcement learning methods in simulation to real-world problems [12] such as self-driving cars and recommendation systems. The casual bandit models the stochastic environment with a causal graph that connects a set of random variables of interest, and actions are modeled by interventions on different subsets of nodes. It provides a critical interpretation of causal influences among random variables [28] that allows us to reason about what will happen to the environment after certain interventions are made to the data-generating process. Compared to the classic MAB with only reward feedback, the casual bandit represents a more detailed model of real-world decision-making problems, including richer feedback from multiple random variables and the causal relations among them. By applying causal reasoning, the learning process can be accelerated and more rewarding interventions can be selected. In applications where the space of intervention is large and explicit experiments are costly, such as social science or economics, it has special value.

Due to the reasons mentioned above, there has been a recent surge of interest in the research community on causal decision-making. Some early attempts include [6], in which the authors took a causal approach to study how to make final decisions based on natural predilections, and proposed an algorithm by modifying Thompson sampling [33]. In [19], the investigators started from a parallel causal structure and extended the algorithm to the general causal bandit problem. Subsequent work [37] took preprocessing steps to get rough estimates of interventional distributions and use them to design efficient experiments to discover the best intervention. In [1], the Bayesian optimization technique is employed to solve the same problem.

The most closely related works focusing on regret minimization in the structural causal bandit with confounders are [22, 23]. In both works, the authors took a graphical characterization using do-calculus [28] to reduce the large intervention set to the possibly optimal minimal intervention set and run a KL-UCB algorithm [14] within the reduced action space. The later work is an extension of the previous one that includes non-manipulable variables. Other lines of causal bandit research normally assume additional prior information, such as infinite observational data [39] or marginal interventional distributions [24, 8]. A general linear model is adopted in [13], which is different from the non-parametric setup in this paper.

In this paper, we focus on the MAB sequential decision-making problem where the arms correspond to interventions on an arbitrary causal graph, including unobserved confounders affecting multiple pairs of variables (also known in causal terminology as the semi-Markovian setting). We recognize the structural causal bandit falls into the category of structured bandit [10, 35]. It has been noted the classical upper confidence bound (UCB) algorithm and Thompson sampling may not fully leverage structural information [31, 20]. It is worth considering the potential limitations of algorithms based on these ideas, such as CRM-AL [25] and generalized Thompson sampling [27]. To address this issue and make systematic use of a known causal structure, we design an algorithm that can meticulously utilize causal structural information. In particular, we make the following contributions:

We provide a logarithmic asymptotic regret lower bound for the structural causal bandit problem with latent confounders by leveraging the canonical SCM formulation [40] to create a space of all possible interventional distributions given a causal graph.

Inspired by the lower bound, we extend [22] and apply approximate allocation matching to design an algorithm that can effectively utilize causal information provided by the causal graph to accelerate the learning process and take informative and rewarding interventions.

We analyze the algorithm to provide a problem-dependent logarithmic upper bound on expected regret and compare it with [22] to show it has a better theoretical guarantee.

We complement the theoretical results by evaluating the proposed approach in a variety of experimental settings featuring different causal structures. We show that our algorithm outperforms the existing baselines in terms of empirical regret.

2 Problem Statement and Background

In this section, we present the formulation of the structural causal bandit problem with latent confounders. We follow the convention to represent a random variable and its value in a capital letter and a lowercase letter respectively. A multivariate random variable is represented in a bold letter.

2.1 Structural Causal Model

Our approach adopts the structural causal model (SCM) [29] to provide a causal perspective on the data-generating process. This allows us to express the relationship between variables and capture important causal concepts, including unobserved variables, observational distributions, and interventional distributions. For a random variable X , let Ω( X ) denote its domain.

Definition 1 (SCM) . A structural causal model is a 4 -tuple ⟨ U , V , F , P ( U ) ⟩ where:

U is a set of unobserved independent background variables (also called exogenous), that determine the randomness of the model.

V := { V i | i = 1 , . . . , n } is the set of observable variables (also called endogenous).

F is a set of structural equations { f 1 , f 2 , ..., f n } such that for each V i ∈ V , f i : Ω( U i ) × Ω( Pa i ) → Ω( V i ) defines a mapping, where U i ⊆ U and Pa i ⊆ V is the parent set of V i . So F as a whole determines a mapping relationship from Ω( U ) to Ω( V ) .

P ( U ) is a joint probability distribution over all the exogenous variables.

̸

Each structural causal model is associated with a causal graph G = ⟨ V , E , B⟩ , where the node set V corresponds to the set of observable variables, E is the set of directed edges and B is the set of bidirected edges. For each V j ∈ Pa i , there exists a directed edge in E such that V j → V i which indicates functional dependency from V j to V i . Without showing unobserved exogenous variable U explicitly, the confounding effects of U are represented using bidirected edges in B , which connect multiple pairs of observable variables in V . The presence of V i ↔ V j in G represents unmeasured factors (or confounders) that may influence both V i and V j . In other words, f i and f j share common exogenous variables U i ∩ U j = ∅ as input. Let | Ω( V ) | be the cardinality of Ω( V ) .

Assumption 1. We assume each V ∈ V can only take a finite number of values, i.e., | Ω( V ) | is finite.

Within the causal graph, an intervention on a subset of random variables S ⊆ V denoted by the do-operator do ( S = s ) sets the structural equation for each S j ∈ S to be S j = s j . We also refer to s as intervention for brevity. If a node V i / ∈ S , its structural equation remains to be V i = f i ( U i , Pa i ) . The empty intervention denoted by do ( ∅ ) does not change the structural equation of any random variable, and the distribution of V is also called observational distribution. Let ✶ denote the indicator function. An SCM induces interventional distributions: if v is consistent with intervention s ,

Otherwise, P ( v | do ( S = s )) = 0 . For a more detailed discussion on SCMs, we refer to [29].

2.2 Causal Bandit Problem with Confounders

The causal bandit problem with confounders studies the sequential decision-making problem with causal information provided by a causal graph G = ⟨ V , E , B⟩ with confounding effects represented by bidirected edge set B . The causal graph G is given but the exact interventional and observational distributions are unknown. Without loss of generality, we assume the reward is collected from node V n , which is bounded in [0 , 1] and can not be intervened on. Let I ⊆ { s ∈ Ω( S ) | S ⊆ V \ V n } be the space of allowed interventions. The decision-maker is required to decide which intervention to apply

at each time step. The expected reward of intervention s ∈ I is µ s = E P s [ V n ] = ∑ v ∈ Ω( V ) v n P s ( v ) , where P s is defined in (1) and E P means the expectation is computed with probability distribution P .

At each time t ∈ { 1 , . . . , T } , the agent follows an adaptive allocation policy π to select an intervention s t ∈ I and observe the causal bandit feedback V t ∼ P s t ( V ) , which is a realization of all observable variables after intervention. In fact, policy π is a sequence { π t } t ∈ N , where each π t determines the probability distribution of taking intervention S t ∈ I given intervention and observation history π t ( S t | s 1 , v 1 , . . . , s t -1 , v t -1 ) . Let µ ∗ := max s ∈I µ s and ∆ s = µ ∗ -µ s denote the optimal mean reward and the expected optimality gap of intervention do ( S = s ) respectively. Given a causal graph G , the objective of the causal bandit problem is to design a policy π to maximize the expected cumulative reward, or equivalently, to minimize the expected regret :

where N T ( s ) is the total number of times intervention do ( S = s ) is taken by policy π until time horizon till T , and the expectation is computed over different realizations of { S t , V t } T t =1 from the interactions between the random policy π and the causal bandit model ⟨ P s ⟩ s ∈I . Thus, R π T is the gap between the expected cumulative rewards of selecting the best intervention all the time and that by the policy π . The regret decomposition in (2) is from [18], and it expresses the expected regret with products of the expected number of suboptimal action selections and expected reward gaps.

The large action space in causal bandits poses many challenges to the learning problem. It is desirable to reduce the action space without excluding all optimal interventions. The do-calculus introduced in [28] provides a set of guidelines for evaluating invariances across interventions. In this context, our specific focus is on Rule 3, which provides the conditions under which a series of interventions have no impact on the outcome variable. For example, Rule 3 implies that P W , Z ( y ) = P Z ( y ) if we have ( Y ⊥ ⊥ W | Z ) G W ∪ Z , where G W ∪ Z is the subgraph of DAG G with incoming edges to the set W ∪ Z removed. This concept leads to the notion of a minimal intervention set (MIS), which is a subset of variables S ⊆ V \ V n such that there is no S ′ ⊂ S for which µ s [ S ′ ] = µ s holds for every Structural Causal Model (SCM) with a causal graph G [22]. Here, s [ S ′ ] denotes an intervention on S ′ ∩ S with values consistent with s . An MIS S is considered a possibly optimal MIS (POMIS) if some intervention s ∈ Ω( S ) can achieve the optimal mean reward in an SCM with a causal graph G . We define I = { s ∈ Ω( S ) | S is a POMIS } .

At an intuitive level, it may seem logical that the most effective course of action would be to intervene on the immediate causes (parents) of the reward variable V n . This approach would provide a higher level of control over V n within the system. If the reward variable V n is not confounded with any of its ancestors, its parent set Pa n is the only POMIS. In more general causal graphs where V n is confounded with any of its ancestors, the paper [22] proves that multiple POMISs exist, which can include variables that are not parents of V n . The paper also provides graphical criteria and an efficient algorithm for constructing a set of all POMISs for a given causal graph. For an effective method to search for I , we refer to [22] and rely on their graphical criteria to construct a set of POMISs. This set of POMISs is used to form the set of possible optimal actions I for use in the causal bandit algorithm. We make the following assumption, which we expect to hold in a general scenario [38], except for certain specifically designed SCMs.

Assumption 2. Within interventions on POMISs, there exists a unique optimal intervention s ∗ .

3 Response Variables and Space of Interventional Distribution Tuples

The unspecified domain of U makes it inconvenient to be directly applied to the learning problem. However, it is noted in [29, Ch. 8] that if each variable in V takes finite states, Ω( U ) can be partitioned and U can be projected to a collection of finite-state response variables. The resulting model is equivalent to the original one with respect to all observational and interventional distributions. Such a technique was used to bound the causal and counterfactual effects with observational distribution [29, Ch. 8]. We take it to define a parameterized space of interventional distribution tuples.

V

1

V

3

V

2

V

4

(a) Causal graph

1

V

1

M

1

V

3

M

3

V

2

M

2

V

4

M

4

(b) Structural equivalent of (a)

V

1

V

2

V

1

M

1

V

2

M

2

(c) Bow graph

3.1 Response Variables and Canonical Structural Causal Model

In Definition 1, each structural equation V i = f i ( U i , Pa i ) connects random variable V i to its parents Pa i . As U i varies along its domain, regardless of how complex the variation is, the only effect it can have is to switch the relationship between Pa i and V i . Since there are at most | Ω( V i ) | | Ω( Pa i ) | mapping relationships from Pa i to V i , we can decompose f i as follows,

where M i ∈ { 0 , . . . , | Ω( V i ) | | Ω( Pa i ) | -1 } is a random variable corresponds to a mapping from Ω( Pa i ) to Ω( V i ) . For an observable variable without parents, we simply have M i ∈ { 0 , . . . , | Ω( V i ) | -1 } . Such a random variable M i ∈ M attached to each V i ∈ V is called response variable in [5]. Treating response variables as exogenous variables, the resulting SCM ⟨ M , V , F , P ( M ) ⟩ is called canonical SCM in [40], where F = { f 1 , f 2 , ..., f n } .

̸

The decomposition of structural equations in equation (3) can be represented graphically by including response variables in M . Since V i = f i ( M i , Pa i ) , for each V i there is a directed edge M i → V i . We also know that if there exists V i ↔ V j in G , both V i and V j are affected by common exogenous variables U i ∩ U j = ∅ . As a result, M i and M j are correlated and they are also connected by a bidirected edge. For example, the causal graph in Fig. 1(a) has a structural equivalent in Fig. 1(b).

Given intervention s ∈ I , there exists a deterministic relationship from M to V . To further explain the idea, consider the bow graph example in Fig. 1(c), where V 1 , V 2 are binary variables taking value 0 or 1 . As illustrated above, the response variables are M 1 ∈ { 0 , 1 } and M 2 ∈ { 0 , 1 , 2 , 3 } , and the structural equations for V 1 and V 2 are as follows:

If do ( ∅ ) is applied, ( V 1 , V 2 ) = (0 , 1) when ( M 1 , M 2 ) = (0 , 2) or (0 , 3) . If do ( V 1 = 1) is applied, ( V 1 , V 2 ) = (1 , 0) when M 1 = 0 or 1 and M 2 = 0 or 2 . It shows that for a given intervention do ( ∅ ) or do ( V 1 = 1) , there exists a deterministic mapping from M to V . Besides, the true value of M can only be inferred from V since M is unobservable. Since M 1 and M 2 are correlated, we define their joint distribution as P ( M 1 = i, M 2 = j ) = p ij . We can further express the observational and interventional probabilities of V with the probability of M as follows,

It can be seen these parameters are sufficient for specifying the model of Fig. 1(c). Such a parameterization technique can be extended to general causal graphs with latent confounders [40].

3.2 Space of Interventional Distribution Tuples

In a causal graph, a confounded component (c-component) is a maximal set of vertices connected with bidirected edges [34]. Note that a singleton node without bidirected edges is also a c-component. For example, there are two c-components in Fig. 1(a): { V 1 , V 3 , V 4 } and { V 2 } . In fact for any causal graph G , its observable variables V can be uniquely partitioned into c-components { V 1 , . . . , V n c ( G ) } , where n c ( G ) is the total number of c-components in G . Then M can also be partitioned into { M 1 , . . . , M n c ( G ) } , where each M j contains response variables adjacent to V j . For example in Fig. 1(b), M is partitioned into { M 1 , M 3 , M 4 } and { M 2 } . Within each M j , the response variables are correlated since they are connected by bidirected edges. Besides, M 1 , . . . , M n c ( G ) are mutually independent since M i and M j are not connected with bidirected edges for any i = j . As a result, P ( m ) = ∏ n c ( G ) j =1 P ( m j ) . By concatenating P ( m j ) for each m j ∈ Ω( M j ) , we construct a vector p j ∈ ∆( | Ω( M j ) | ) , where ∆( | Ω( M j ) | ) := { p ′ j ∈ R | Ω( M j ) | ≥ 0 | 1 ⊤ p ′ j = 1 } and 1 ⊤ is the transpose of the all-ones vector.

̸

Let the parent set of a c-component V j be Pa V j := ( ∪ i : V i ∈ V j Pa i ) \ V j . When taking intervention do ( S = s ) , the values of V j ∩ S is set to s [ V j ] , which denotes the values of V j ∩ S that is consistent with s . Notice that M j picks the mapping functions from Pa i to V i for all V i ∈ V j . Seeing values v j , pa V j and s [ V j ] , there exists a set of configurations of M j , denoted by B G , s [ V j ] ( v j , pa V j ) ⊆ Ω( M j ) , that can make this happen. By marking configurations in B G , s [ V j ] ( v j , pa V j ) with 1 and others to be 0 , we construct a vector b G , s [ V j ] ( v j , pa V j ) ∈ { 0 , 1 } | Ω( M j ) | . With M 1 , . . . , M n c ( G )

being mutually independent, the interventional distribution can be factorized as

In the bow graph example in Fig. 1(c), there is one c-component, and vector b ⊤ G , s [ V j ] ( v j , pa V j ) can be constructed by referring to (4), where p 1 is a concatenation of p i,j . The result in (5) generalizes the parameterization for the bow graph to a general causal graph. Based on it, given a causal graph G , we define the space of interventional distribution tuples as the following.

Definition 2 (Space of Interventional Distribution Tuples) . Given a causal graph G = ⟨ V , E , B⟩ , the space of interventional (and observational) distribution tuples P = ⟨ P s ⟩ s ∈I is

Space M G contains all interventional distribution tuples associated with canonical SCMs ⟨ M , V , F , P ( M ) ⟩ with arbitrary P ( M ) . Since the space of { p 1 , . . . , p n c ( G ) } is compact and a continuous image of a compact space is compact, M G is also a compact space.

4 An Asymptotic Regret Lower Bound

In this section, we present an asymptotic information-theoretic regret lower bound for the causal bandit problem with confounders. It quantifies the optimal asymptotic performance of a uniformly good causal bandit policy defined below.

Definition 3 (Uniformly Good Policy) . Given a causal graph G = ⟨ V , E , B⟩ , a causal bandit policy π is uniformly good if for any α > 0 , the expected regret of π for any interventional distribution tuple setup P ∈ M G , denoted by R π T ( P ) , satisfies

A policy π being uniformly good indicates that it can achieve subpolynomial regret for any P ∈ M G . The information-theoretic regret lower bound sets the limit for the asymptotic regret of all such policies. Let D ( P ∥ Q ) denote the Kullback-Leibler (KL) divergence of two probability distributions P and Q and let I -s = I \ { s } .

Theorem 1. Given a causal graph G = ⟨ V , E , B⟩ , let P = ⟨ P s ⟩ s ∈I ∈ M G be the true interventional distribution tuple with a unique optimal intervention s ∗ ∈ I with mean reward µ ∗ . The expected regret for any uniformly good causal bandit policy π satisfies

where C ( P , G ) is the value of the optimization problem given below,

where µ ′ s is the expected reward of intervention s according to probability distribution P ′ s .

Theorem 1 can be viewed as an extension of [10, Th. 1], and we defer its proof to appendices. The optimization problem O ( P, M G ) is a semi-infinite program since there are infinite P ′ in (7). To interpret the lower bound, (6) is a minimization of the regret, and its solution indicates an optimal allocation of O (ln T ) number of explorations, which is η s ln T for each s ∈ I -s ∗ as T →∞ . In (7), ∑ s ∈I η s D ( P s ∥ P ′′ s ) can be viewed as the distance generated between true interventional distribution tuple P and an alternative P ′′ ∈ M G by an exploration allocation strategy ⟨ η s ⟩ s ∈I -s ∗ . For any P ′ in (7), µ ′ s ∗ = µ ∗ since P ′ s ∗ = P s ∗ , and as a result, max s ∈I -s ∗ µ ′ s ≤ µ ∗ indicates intervention s ∗ is also optimal in P ′ . So (7) sets a constraint for ⟨ η s ⟩ s ∈I -s ∗ : the models allowed to have distance < 1 from the true model must also take s ∗ to be optimal. Or conversely, ⟨ η s ⟩ s ∈I -s ∗ must generate distance ≥ 1 for any P ′ ∈ M G with P ′ s ∗ = P s ∗ not taking intervention s ∗ to be optimal.

Remark 1. The O (ln T ) regret lower bound holds only if there exists P ′ ∈ M G such that P ′ s ∗ = P s ∗ , which is a condition meaning P can not be distinguished from all other interventional distribution tuples in M G by only taking intervention s ∗ . This condition holds for almost all causal bandit problems. Otherwise, constraint (7) is relaxed, and the value of the optimization problem C ( P , M G ) = 0 , which indicates sub-logarithmic regret can be achieved. Following the same line of proof in [17, Th. 3.9], it can be shown simple UCB can achieve this sub-logarithmic, in fact constant, expected regret.

5 SCM-based Causal Bandit Algorithm

The regret lower bound in Theorem 1 suggests a potentially optimal exploration strategy, i.e. to take each intervention s ∈ I -s ∗ up to η s ln T times. However, it requires the ground truth P to solve the optimization problem O ( P , G ) . The allocation matching principle [10] essentially replaces the true model with an estimated one to solve the optimization problem, and controls explorations to match with the solution. It is expected that the estimated model converges to the ground truth at a fast enough rate so that the actual explorations match with the optimal exploration strategy. We follow this idea to design a causal bandit policy, whose pseudo-code is shown in Algorithm 1. We call this SCM-based Approximate Allocation Matching (SCM-AAM).

Similar to [22], SCM-AAM only intervenes on POMISs for the purpose of reducing the action space. The algorithm is anytime without requiring the knowledge of horizon length. It takes two tuning parameters ϵ = (0 , 1 / |I| ) and γ > 0 as inputs that control the exploration rate, which affects finite-time performance. At each time t , let N t ( s ) be the number of times the intervention do ( S = s ) is taken so far, and let N t ( v , s ) be the number of times we observe v with intervention do ( S = s ) . The algorithm maintains a set of empirical interventional distributions ⟨ ¯ P s ,t ⟩ s ∈I and empirical mean rewards ⟨ ¯ µ s ,t ⟩ s ∈I , where ¯ P s ,t ( v ) = N t ( v , s ) /N t ( s ) for each s ∈ I .

The main body of the algorithm is composed of two components: exploitation and exploration. At each round t , it first attempts to evaluate if enough information has been collected to determine the optimal action. Inspired by the lower bound, enough distance is generated by the sampling history between ⟨ ¯ P s ,t ⟩ s ∈I and P ′ = ⟨ P ′ s ⟩ s ∈I if ∑ s ∈I N t ( s ) D ( ¯ P s ,t ∥ P ′ s ) > (1 + γ ) ln t . So we can construct a confidence set C t with high probability to contain the true model P , which is defined as

1

2

3

Algorithm 1: SCM-based Approximate Allocation Matching (SCM-AAM)

Input

: Causal graph G = ⟨ V , E , B⟩ , I = { s ∈ Ω( S ) | S is POMIS }

Initialization :

N e t = 0 , ∀ s ∈ I : N t ( s ) = 0

Set :

ϵ ∈ (0 , 1 / |I| ) and γ > 0

Output

: Sequence of interventions

Select each intervention s ∈ I once for t = |I| +1 to T do

Compute empirical distributions and mean rewards

With

⟨

defined in (8), compute UCB

if

s

t

s

>

˜

µ

(

)

,

C

∀

∈ I

∃

C

t

s

,t

∗

: ¯

µ

Select

s

∗

S

else if min

Select

,t

t

∈I

s

S

Set

N

t

N

t

e

else

Select

Set

N

S

e

t

N

t

N

, and

N

s

∗

,t

t

(

)

ϵN

s

arg min

e

-

t

e

t

∈I

s

t

+1

and

t

S

(

)

then

N

t

(

s

N

e

t

+1

s

∈I

and

t

(

S

˜

µ

N

t

(

t

s

)

s

∗

,t

˜

µ

s

(

C

then

N

t

t

S

(

) + 1

%exploitation

%forced exploration

)

(

S

C

t

t

)

N

t

(

S

t

) + 1

)

%exploration with greedy approximation

N

t

(

S

t

) + 1

With V n being the reward node, for each s ∈ I , the algorithm computes the UCB of the mean reward of taking intervention s as ˜ µ s ( C t ) = ∑ v ∈ Ω( V ) v n max P ′ ∈C t P ′ s ( v ) . If there exists an intervention s ∗ ,t with empirical mean reward ¯ µ s ∗ ,t > ˜ µ s ( C t ) , ∀ s ∈ I -s ∗ ,t , we are confident about s ∗ ,t to be the optimal intervention. So the algorithm enters the exploitation phase and selects S t = s ∗ ,t .

If no such s ∗ ,t exists, the algorithm needs to explore. According to the allocation principle, one is inclined to solve O ( ⟨ ¯ P s ,t ⟩ s ∈I , G ) defined in Theorem 1 and make N t ( s ) ≈ η s ln t for each s ∈ I . Nevertheless, there is no guarantee that this semi-infinite program can be solved efficiently. Therefore, we take a greedy approximation approach. Ideally, to get out of exploration, the algorithm needs to reduce max s ∈I -s ∗ ˜ µ s ( C t ) to a value below µ ∗ , thus it selects S t = arg max s ∈I ˜ µ s ( C t ) . The algorithm maintains a counter of exploration steps N e t . If min s ∈I N t ( s ) ≤ ϵN e t , it generates forced exploration by selecting the least selected intervention S t . Such a mechanism ensures every action is persistently taken in exploration phases so that ⟨ ¯ P s ,t ⟩ s ∈I converges to the ground truth eventually.

Remark 2. With factorization in (5) , C t corresponds to setting constraint for { p 1 , . . . , p n c ( G ) } as

where the expression on the left of inequality is a convex function of p 1 , . . . , p n c ( G ) . Define convex set C p t = { { p 1 , . . . , p n c ( G ) } | p j ∈ ∆( | Ω( M j ) | ) , ∀ j ∈ { 1 , . . . , n c ( G ) } , and (9) is true } . Then the UCB can be derived by maximizing a set of concave functions over C p t as follows,

We use ˜ µ s ( C t ) instead of ˜ µ ′ s ( C t ) = max P ′ ∈C t ∑ v ∈ Ω( V ) v n P ′ s ( v ) as UCB since ∑ v ∈ Ω( V ) v n P ′ s ( v ) in general is a non-concave function of p 1 , . . . , p n c ( G ) . Also, notice that ∀ s ∈ I : ˜ µ s ( C t ) ≥ ˜ µ ′ s ( C t ) .

In a pure topology-based approach [22], each intervention on POMISs is treated independently during the algorithm execution. Whereas SCM-AAM can leverage the canonical SCM to incorporate the sampling results from different interventions together. After each new sample v t with intervention s t is observed, it is used to update the parametric space for interventional distribution tuples C t .

v

P

s

,t

¯

⟩

s

∈I

and

⟨

∈

Ω(

V

)

v

n

¯

µ

P

s

,t

′

∈C

⟩

t

t

) =

∑

s

∈I

P

′

s

(

v

)

,

∀

s

∈ I

6 Finite Time Regret Analysis

We present a finite time problem-dependent regret upper bound for SCM-AAM in Theorem 2. The complete proof is provided in the appendices. We also give an interpretation of Theorem 2, and compare the regret upper bound with [22] in the setting where the reward node V n is binary,

Theorem 2. For the causal bandit problem with unobserved confounders, suppose the causal graph G = ⟨ V , E , B⟩ is given and Assumptions 1 and 2 are true. For any interventional distribution tuple P ∈ M G , any κ > 0 and horizon of length T , the expected regret for SCM-AAM with parameters ϵ ∈ (0 , 1 / |I| ) and γ > 0 satisfies

where c is a suitably large universal constant depending on P , κ and tuning parameters ϵ and γ , and

Theorem 2 indicates the expected number of samples for each s ∈ I -s ∗ can be approximately bounded by ζ s ( P ) ln T . It can be interpreted as follows: when s ∈ I -s ∗ is selected approximately up to ζ s ( P ) ln T times, its UCB can be pushed below µ ∗ . The presence of terms containing ϵ is due to forced exploration. In [22], KL-UCB and Thompson sampling are employed to apply interventions on POMISs. If the reward node V n ∈ { 0 , 1 } , they enjoys expected regret ∑ s ∈I -s ∗ (∆ s ln T ) /D ( P s ( V n ) ∥ P s ∗ ( V n )) + c ′ for some insignificant c ′ [14, 2]. In this situation, Theorem 2 shows that the SCM-AAM enjoys smaller expected regret when we disregard the small constants κ , γ and ϵ . To see it, we express the leading constant for KL-UCB and Thompson as

where C ′ s ( η s , P ( V n )) = { P ′ s ( V n ) ∈ [0 , 1] | η s D ( P s ( V n ) ∥ P ′ s ( V n )) ≤ 1 } . Note that C ′ s ( η s , P ( V n )) only sets constraint on P ′ s ( V n ) , while C s ( η s , P ) sets constraints on P ′ s ( V ) . As a result, C s ( η s , P ) ⊆ C ′ s ( η s , P ( V n )) , so that ζ s ( P ) ≤ 1 /D ( P s ( V n ) ∥ P s ∗ ( V n )) .

7 Experimental Results

We compare the empirical performance of SCM-AAM with existing algorithms. The first baseline algorithm employs the simple UCB algorithm [4] on a reduced action set that includes interventions on POMISs. The other two baselines as introduced in [22], utilize the KL-UCB algorithm and Thomposn sampling (TS) to intervene on POMISs. We compare the performance of all the algorithms on three different causal bandit instances shown in Fig. 2. We choose the input parameters of the SCM-AAM algorithm to be γ = 0 . 1 and ϵ = 1 / | 5 I| in the simulations. We set the horizon to 800 for all three tasks and repeat every simulation 100 times. Additionally, we include a confidence interval around the mean regret, with the width of the interval set to twice the standard deviation. The structural equations for the three causal bandit instances, as well as experimental details, can be found in the appendices. We plot the mean regret against time, where time in this context corresponds to the number of actions. All the nodes in Fig. 2 take binary values, either 0 or 1 .

For task 1, we consider the causal graph shown in Fig. 2(a) with POMISs { V 1 } and { V 2 } . For task 2 and task 3, we consider more complex causal graphs, as displayed in Fig. 2(b) and (c). The POMISs for task 2 include { V 1 } , { V 3 } , and { V 3 , V 4 } , while for task 3, they are { V 1 , V 4 , V 6 } and { V 5 , V 6 } . The UCB algorithm has inferior performance compared to other baseline algorithms, especially in the more complicated tasks 2 and 3 . The TS algorithm outperforms both UCB and KL-UCB in all three settings; however, it still incurs more regret than our proposed algorithm. The experiments demonstrate that our proposed SCM-AAM algorithm consistently outperforms other baseline algorithms by incurring lower empirical regret across all three tasks. Notably, the performance advantage of SCM-AAM is more significant for tasks 2 and 3, which have a more intricate causal graph structure with a higher number of nodes and edges. The results indicate that

V

2

V

1

V

3

(a) Task

1

V

1

V

2

V

3

V

4

V

5

(b) Task

2

V

1

V

3

V

2

V

4

V

5

V

6

V

7

(c) Task

3

(a) Task

1

(b) Task

2

(c) Task

3

100

SCM-AAM

100

SCM-AAM

100

KL-UCBonPOMISs

△SCM-AAM

80

KL-UCB onPOMISs

OUCBonPOMISs

80

-OUCBonPOMISs

KL-UCBonPOMISs

80

-UCB on POMISs

←TSonPOMISs

←TSonPOMISs

TS on POMISs

60

60

Regret

60

40

40

40

20

20

20

0

0

200

400

600

800

0

0

200

400

600

800

0

0

200

400

600

800

Time

Time

Time

incorporating causal structural information through the SCM-AAM algorithm can significantly reduce expected regret, especially for more complex causal graphs.

The simulations were conducted on a Windows desktop computer featuring a 12 th generation Intel Core i 7 processor operating at 3 . 1 GHz and 32 GBof RAM. No GPUs were utilized in the simulations. The runtime for each iteration of the SCM-MAB, involving 800 arm plays, was approximately 2 minutes for both Task 1 . However, for Task 2 and Task 3 , which encompass larger underlying causal graphs, the runtime for every iteration increased to around 20 minutes. The algorithm code is provided at https://github.com/CausalML-Lab/SCM-AAM.

8 Conclusion

In this paper, we studied the causal bandit problem with latent confounders. With causal information provided by a causal graph, we present a problem-dependent information-theoretic regret lower bound. Leveraging the canonical SCM, we take an approximate allocation matching strategy to design the SCM-AAM algorithm. By analyzing SCM-AAM, we show it has a problem-dependent logarithmic regret upper bound. The analytic result is complemented with numerical illustrations featuring a variety of causal structures. It is shown that SCM-AAM exhibits promising performance in comparison with classic baseline algorithms. Since SCM has a natural application in counterfactual reasoning, extending the proposed algorithm and theoretical results to a counterfactual decisionmaking setup is an interesting future direction.

References

V. Aglietti, X. Lu, A. Paleyes, and J. González. Causal bayesian optimization. In International Conference on Artificial Intelligence and Statistics , pages 3155-3164. PMLR, 2020.

S. Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

J. M. Antle and S. M. Capalbo. Econometric-process models for integrated assessment of agricultural production systems. American Journal of Agricultural Economics , 83(2):389-401, 2001.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

A. Balke and J. Pearl. Counterfactual probabilities: Computational methods, bounds and applications. In Uncertainty Proceedings 1994 , pages 46-54. Elsevier, 1994.

E. Bareinboim, A. Forney, and J. Pearl. Bandits with unobserved confounders: A causal approach. Advances in Neural Information Processing Systems , 28, 2015.

J. Bartroff, T. L. Lai, and M.-C. Shih. Sequential experimentation in clinical trials: design and analysis , volume 298. Springer Science & Business Media, 2012.

B. Bilodeau, L. Wang, and D. M. Roy. Adaptively exploiting d-separators with causal bandits. arXiv preprint arXiv:2202.05100 , 2022.

R. Combes and A. Proutiere. Unimodal bandits: Regret lower bounds and optimal algorithms. In International Conference on Machine Learning , pages 521-529. PMLR, 2014.

R. Combes, S. Magureanu, and A. Proutiere. Minimal exploration in structured stochastic bandits. Advances in Neural Information Processing Systems , 30, 2017.

A. D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research , 23(1):10237-10297, 2022.

G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning , 110(9):2419-2468, 2021.

S. Feng and W. Chen. Combinatorial causal bandits. arXiv preprint arXiv:2206.01995 , 2022.

A. Garivier and O. Cappé. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory , pages 359-376. JMLR Workshop and Conference Proceedings, 2011.

T. L. Graves and T. L. Lai. Asymptotically efficient adaptive choice of control laws incontrolled markov chains. SIAM journal on control and optimization , 35(3):715-743, 1997.

A. H. Guide. Infinite dimensional analysis . Springer, 2006.

B. Hao, T. Lattimore, and C. Szepesvari. Adaptive exploration in linear contextual bandit. In International Conference on Artificial Intelligence and Statistics , pages 3536-3545. PMLR, 2020.

T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

F. Lattimore, T. Lattimore, and M. D. Reid. Causal bandits: Learning good interventions via causal inference. Advances in Neural Information Processing Systems , 29, 2016.

T. Lattimore and C. Szepesvari. The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics , pages 728-737. PMLR, 2017.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

S. Lee and E. Bareinboim. Structural causal bandits: where to intervene? Advances in Neural Information Processing Systems , 31, 2018.

S. Lee and E. Bareinboim. Structural causal bandits with non-manipulable variables. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4164-4172, 2019.

Y. Lu, A. Meisami, A. Tewari, and W. Yan. Regret analysis of bandit problems with causal background knowledge. In Conference on Uncertainty in Artificial Intelligence , pages 141-150. PMLR, 2020.

A. Maiti, V. Nair, and G. Sinha. A causal bandit approach to learning good atomic interventions in presence of unobserved confounders. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

J. S. McMullen. Entrepreneurial judgment as empathic accuracy: A sequential decision-making approach to entrepreneurial action. Journal of Institutional Economics , 11(3):651-681, 2015.

P. A. Ortega and D. A. Braun. Generalized thompson sampling for sequential decision-making and causal inference. Complex Adaptive Systems Modeling , 2(1):1-23, 2014.

J. Pearl. Causal diagrams for empirical research. Biometrika , 82(4):669-688, 1995.

J. Pearl. Causality . Cambridge university press, 2009.

H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527-535, 1952.

D. Russo and B. Van Roy. Learning to optimize via information-directed sampling. Operations Research , 66(1):230-252, 2018.

V. Srivastava, P. Reverdy, and N. E. Leonard. Surveillance in an abruptly changing world via multiarmed bandits. In IEEE Conf. on Decision and Control , pages 692-697, Los Angeles, CA, Dec. 2014.

W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294, 1933.

J. Tian and J. Pearl. A general identification condition for causal effects. In Eighteenth national conference on Artificial intelligence , pages 567-573, 2002.

B. P. Van Parys and N. Golrezaei. Optimal learning for structured bandits. arXiv preprint arXiv:2007.07302 , 2020.

C. H. Weiss, M. J. Bucuvalas, and M. Bucuvalas. Social science research and decision-making . Columbia University Press, 1980.

A. Yabe, D. Hatano, H. Sumita, S. Ito, N. Kakimura, T. Fukunaga, and K.-i. Kawarabayashi. Causal bandits with propagating inference. In International Conference on Machine Learning , pages 5512-5520. PMLR, 2018.

P. B. Yale. Geometry and symmetry . Courier Corporation, 2014.

J. Zhang and E. Bareinboim. Transfer learning in multi-armed bandit: a causal approach. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , pages 1778-1780, 2017.

J. Zhang, J. Tian, and E. Bareinboim. Partial counterfactual identification from observational and experimental data. In International Conference on Machine Learning , pages 26548-26558. PMLR, 2022.

Appendices

A Proof of Asymptotic Regret Lower Bound in Theorem 1

The lower bound is derived following the same strategy in [21] by applying divergence decomposition and Bretagnolle-Huber inequality. For completeness, we reproduce both proofs in this section. Readers familiar with these results can skip them. Besides, we note that a more general result on controlled Markov chains is proven in [15].

Recall that a policy π is composed of a sequence { π t } t ∈ N > 0 , where at each time t ∈ { 1 , . . . , T } , π t determines the probability distribution of taking intervention S t ∈ I given intervention and observation history π t ( S t | s 1 , v 1 , . . . , s t -1 , v t -1 ) . So the intervention and observation sequence { S t , V t } t ∈ N > 0 is a production of the interactions between the interventional distribution tuple ⟨ P s ⟩ s ∈I and policy π . Let T be the horizon length, we define a probability measure P on the sequence of outcomes induced by ⟨ P s ⟩ s ∈I and π such that

For a fixed policy π , let P and P ′ be the probability measures on the T -round plays of two different causal bandit instances with different interventional distribution tuples P = ⟨ P s ⟩ s ∈I and P ′ = ⟨ P ′ s ⟩ s ∈I in space M G , which is defined in Definition 2. From (10), we get the distinction between P and P ′ is exclusively due to the separations of P and P ′ . As a result, D ( P ∥ P ′ ) has the following decomposition, which is a standard result in MAB problems [21, Th. 15.1].

Lemma 3 (Divergence Decomposition) . Given a causal graph G = ⟨ V , E , B⟩ , let P = ⟨ P s ⟩ s ∈I be the true interventional distribution tuple and let P ′ = ⟨ P ′ s ⟩ s ∈I be different from P . For some fixed policy π , let P and P ′ be the probability measures on the T rounds of causal bandit plays in P and P ′ respectively. Then,

where the expectation is computed with probability measure P .

Proof. For a fixed policy π , from (10), we get

As a result, π t is reduced and we get

which shows the distinction between P and P ′ is exclusively due to the separations of P s and P ′ s for each s ∈ I . Then we can decompose D ( P ∥ P ′ ) with (11) as the following,

which concludes the proof.

The other tool to prove the regret lower bound is the Bretagnolle-Huber Inequality.

Lemma 4 (Bretagnolle-Huber Inequality [21, Th. 14.2]) . Let P and Q be two probability measures on a measurable space (Ω , F ) , and let E ∈ F be an arbitrary event. Then

where E ∁ = Ω \ E is complement of E .

Proof with finite Ω . In Assumption 1, we assume each node in the causal graph can only take a finite number of values. So the sample space for the causal bandit problem with horizon length T is finite. We provide proof for the Bretagnolle-Huber inequality assuming Ω is finite. First, we show

Applying Cauchy-Schwarz inequality, we get

We conclude our proof.

To prove the regret lower bound, the remaining work regards applying divergence decomposition and substituting D ( P ∥ P ′ ) into the Bretagnolle-Huber inequality.

Lemma 5. Given a causal graph G = ⟨ V , E , B⟩ , let P = ⟨ P s ⟩ s ∈I ∈ M G be the true interventional distribution tuple with a unique optimal intervention s ∗ ∈ I with mean reward µ ∗ . The expected regret for any uniformly good causal bandit policy π satisfies

where C ( P , M G ) is the value of the optimization problem given below

where I ′′ ∗ is the set of optimal interventions according to P ′′ .

Remark 3. To interpret Lemma 5, (12) is a minimize of the regret, and its solution indicates an optimal exploration strategy. Each s ∈ I -s ∗ should be selected up to η s ln T as T →∞ . In (13) , ∑ s ∈I η s D ( P s ∥ P ′′ s ) is viewed as the distance generated between true interventional distribution tuple P and an alternative P ′′ ∈ M G by an exploration strategy ⟨ η s ⟩ s ∈I -s ∗ . So (13) requires ⟨ η s ⟩ s ∈I -s ∗ must generate distance ≥ 1 for any P ′ ∈ M G in which intervention s ∗ is not optimal.

Proof. For any P ′ ∈ {P ′′ ∈ M G | s ∗ / ∈ I ′′ ∗ } , the optimal intervention s ∗ according to ground truth P is not optimal in P ′ . Recall P ′ is the measures on T -round plays of causal bandit with interventional distribution tuple P ′ . Also let µ ′ ∗ be the optimal mean reward and ∆ ′ s = µ ′ ∗ -µ ′ s . For true causal model P , P and ∆ s are defined similarly. For a fixed policy π , let R T and R ′ T be the expected regret from T round plays in P and P ′ respectively. Letting ϵ = min { ∆ s ′ ∗ , ∆ ′ s ∗ } , we get

Combining both inequalities, we get

Applying Bretagnolle-Huber inequality in Lemma 4 by substituting event { N T ( s ∗ ) < T 2 } into E ,

With Lemma 3, we substitute ∑ s ∈I E [ N T ( s )] D ( P s ∥ P ′ s ) into D ( P ∥ P ′ ) and rearrange (14),

According to Definition 3, that for any α > 0 , the regret for a uniformly good policy π satisfies

which sets the constraints for E [ N T ( s )] / ln T as T →∞ . Since R T = ∑ s ∈I -s ∗ E [ N T ( s )]∆ s , by replacing E [ N T ( s )] / ln T with η s , we finish the proof.

̸

Proof of Theorem 1. We finish the proof by showing the equivalence of ˜ O ( P , M G ) in Lemma 5 and O ( P , M G ) in Theorem 1. First notice that there is no η s ∗ in (12). Accordingly, η s ∗ can be assigned to an arbitrarily large value to make ∑ s ∈I η s D ( P s ∥ P ′ s ) ≥ 1 if P s ∗ = P ′ s ∗ . Therefore, (13) is equivalent to

It means if s ∗ is not optimal in P ′ , then ∑ s ∈I -s ∗ η s D ( P s ∥ P ′ s ) ≥ 1 . Its contrapositive statement is that if ∑ s ∈I -s ∗ η s D ( P s ∥ P ′ s ) < 1 , then s ∗ is also optimal in P ′ . Putting it into the mathematical expression, we get

′ ′

where we use the fact µ ∗ = µ ∗ since P s ∗ = P s ∗ . The proof is concluded.

B Finite time Regret Analysis of SCM-AAM in Theorem 2

B.1 Guarantee with Forced Exploration

The SCM-AAM algorithm maintains a counter of exploration steps N e t . In expiration steps, if min s ∈I N t ( s ) ≤ ϵN e t , the SCM-AAM algorithm generates forced exploration by selecting one of the least selected intervention. Lemma 6 shows that the forced exploration mechanism ensures every intervention is persistently selected in exploration phases.

It follows from (15) that

Lemma 6. Let R = { t i } i ≥ 1 be the sequence of rounds that SCM-AAM enters the exploration phase. For any time t i ∈ R , it satisfies that

Proof. We first note that the following two facts are true

min s ∈I N t ( s ) is non-decreasing over t .

If min s ∈I N t i ( s ) ≤ ϵN e t i , then min s ∈I N t i + |I| ( s ) ≥ min N t i ( s ) + 1 .

Since N t ( s ) is non-decreasing over t , the first statement is true. The second statement is true since otherwise, S t is selected by forced exploration at least |I| times without increasing min s ∈I N t ( s ) . With these two facts, we are ready to provide proof by contradiction. Suppose there exists i ∈ { 1 , . . . , |R|} such that

According to the first fact, we have ∀ j ≥ i/ 2 ,

According to SCM-AAM, ϵ ∈ (0 , 1 / |I| ) . Then we apply the second fact,

which creates a contradiction.

B.2 Supporting Lemmas (Concentration Inequalities)

The following concentration inequalities are instrumental for regret analysis. For bandit with feedback drawn from arbitrary discrete distributions, a concentration bound on the empirical distribution is presented in [35, Lemma 6]. In the structured causal bandit problem with unobserved confounders, the actions space is I , and the discrete support of feedback is Ω( V ) . At each time t , for each intervention s ∈ I , recall ¯ P s ,t is the empirical interventional distribution of V and N t ( s ) is the number of times the intervention do ( S = s ) is taken till t . For each intervention s ∈ I , the true interventional distribution is P s .

Lemma 7 (Concentration Inequality for Information Distance [35]) . Let δ ≥ |I| ( | Ω( V ) | -1) . Then for any t > 0 ,

llowing Corollary 8 is a result of Lemma 7.

Corollary 8. For any γ > 0 , there exists suitably large universal constant c such that

Proof. We select t ′ such that (1 + γ ) ln t ′ ≥ |I| ( | Ω( V ) | -1) . We apply Lemma 7 to get

which remains finite. We conclude our proof.

Lemma 9 proposed in [9, Lemma 4.3] extends Hoeffding's inequality to provide an upper bound on the deviation of the empirical mean sampled at a stopping time. The time SCM-AAM enters the exploration phase is a stopping time, and the time each intervention is selected is also a stopping time. It will be used in multiple circumstances in regret analysis.

Lemma 9 (Extension of Hoeffding's Inequality [9]) . Let { Z t } t ∈ N > 0 be a sequence of independent random variables with values in [0 , 1] . Let F t be the σ -algebra such that σ ( Z 1 , . . . , Z t ) ⊂ F t and the filtration F = {F t } t ∈ N > 0 . Consider s ∈ N , and T ∈ N > 0 . We define S t = ∑ t j =1 ϵ j ( Z j -E [ Z j ]) , where ϵ j ∈ { 0 , 1 } is a F j -1 -measurable random variable. Further define N t = ∑ t j =1 ϵ j . Define ϕ ∈ { 1 , . . . , T +1 } a F -stopping time such that either N ϕ ≥ s or ϕ = T +1 . Then we have that

In Corollary 10, we extend Lemma 9 to bound the L 1 deviation of the empirical distribution.

Corollary 10 ( L 1 deviation of the empirical distribution) . Let A denote finite set { 1 , . . . , a } . For two probability distribution Q and Q ′ on A , let ∥ Q ′ -Q ∥ 1 = ∑ a k =1 | Q ′ ( k ) -Q ( k ) | . Let X t ∈ A be a sequence of independent random variables with common distribution Q . Let F t be the σ -algebra such that σ ( X 1 , . . . , X t ) ⊂ F t and the filtration F = {F t } t ∈ N > 0 . Let ϵ t ∈ { 0 , 1 } is a F t -1 -measurable random variable. We define

For s ∈ N , and T ∈ N > 0 , let ϕ ∈ { 1 , . . . , T +1 } be a F -stopping time such that either N ϕ ≥ s or ϕ = T +1 . Then we have

Proof. It is known that for any distribution Q ′ on A ,

Then we apply a union bound to get

which concludes the proof.

B.3 Supporting Lemmas (Continuity of Upper Confidence Bound)

For a constrained optimization problem, Berge's maximum theorem imposes additional restrictions on the objective function and constraint set to guarantee that the problem's solution varies smoothly with parameters. Specifically, it studies the following optimization problem

where F : X × Θ → R the objective function, and G ( θ ) is the set of feasible values for X given θ defined by the multi-valued correspondence G : Θ ⇒ X . We assume that G ( θ ) ⊆ X is compact for all θ ∈ Θ .

As a consequence,

̸

Definition 4. The compact valued correspondence G is continuous at θ ∈ Ω( θ ) if it is both upper and lower hemicontinuous at θ . The upper and lower hemicontinuous can be verified as below.

Upper hemicontinuous: the correspondence G is upper hemicontinuous at θ ∈ Θ if G ( θ ) is nonempty and if, for every sequence { θ j } with θ j → θ and every sequence { x j } with x j ∈ G ( θ j ) for all j , there exists a convergent subsequence { x j k } such that x j k → x ∈ G ( θ ) .

Lower hemicontinuous: the correspondence G is lower hemicontinuous at θ ∈ Θ if G ( θ ) is nonempty and if, for every x ∈ G ( θ ) and every sequence { θ j } such that θ j → θ , there is a J ≥ 1 and a sequence { x j } such that x j ∈ G ( θ j ) for all j ≥ J and x j → x .

The correspondence G is continuous if it is continuous at every θ ∈ Ω( θ ) .

Lemma 11 (Berge's Maximum Theorem [16, Ch. 17.31]) . Let X ⊆ R n and Θ ⊆ R m . Let F : X × Θ → R be a continuous function, and let G : Θ → X be a compact valued and continuous correspondence. Then the maximum value function

is well-defined and continuous, and the optimal correspondence

is nonempty, compact valued, and upper hemicontinuous.

We want to leverage Berge's maximum theorem to prove the UCB index ˜ µ s ( C s ( η s , P )) = ∑ v ∈ Ω( V ) v n max P ′ ∈C s ( η s , P ) P ′ s ( v ) is continuous with respect to η s and P . Recall in Theorem 2, for given interventional distribution tuple P ∈ M G and η s > 0 , C s ( η s , P ) is the set of feasible interventional distribution tuples P ′ , and it is defined as

Lemma 12. For any s ∈ I , the correspondence C s : R ≥ 0 ×M G ⇒ M G is a compacted valued continuous correspondence.

Proof. The space of interventional distribution tuples M G is compacted as has been explained below Definition 2. Besides, for any P ∈ M G and η s ≥ 0 , the constrain η s D ( P s ∥ P ′ s ) + ϵη s / 2 ∑ x ∈I -s D ( P x ∥ P ′ x ) ≤ 1 defines a closed compact set for P ′ = ⟨ P ′ s ⟩ s ∈I . Since the intersection of two compact sets is compact if one of them is also closed, C s ( η s , P ) is compact. It remains to show C s is both upper hemicontinuous and lower hemicontinuous.

Upper hemicontinuous: Fix arbituary η s ≥ 0 and P ∈ M G , and let { ( η s ,i , P i ) } i ∈ N > 0 be a sequence such that η s ,i ≥ 0 , P i ∈ M G , and ( η s ,i , P i ) → ( η s , P ) . Notice that each C s ( η s ,i , P i ) is non-empty since P i ∈ C s ( η s ,i , P i ) . Therefore, there is a sequence {P ′ i } i ∈ N > 0 such that P ′ i ∈ C s ( η s ,i , P i ) . Since η s ,i → η s , there exists a closed and bounded set Θ ⊂ R ≥ 0 such that η s ∈ Θ , and for some N ≥ 1 , η s ,i ∈ Θ for all i ≥ N . Moreover, P i and P ′ i are also bounded since every interventional probability tuple in M G is bounded. Since each ( η s ,i , P i , P ′ i , ) lies in a closed and bounded subset for i ≥ N , by Bolzano-Weierstrass theorem, the sequence { ( η s ,i , P i , P ′ i , ) } i ∈ N > 0 has a convergent subsequence { ( η s ,i k , P i k , P ′ i k ) } k ∈ N > 0 with limit point ( η s , P , P ′ ) . Since each element of this sequence satisfies

the inequality holds at the limit point ( η s , P ′ ) , i.e.,

due to that h is a continuous function. We get C s is upper hemicontinuous.

Lower hemicontinuous: For arbituary η s ≥ 0 and P ∈ M G , we fix P ′ ∈ C s ( η s , P ) . Let { ( η s ,i , P i ) } i ∈ N > 0 be a sequence such that η s ,i ≥ 0 , P i = ⟨ P s ,i ⟩ s ∈I ∈ M G , and ( η s ,i , P i ) → ( η s , P ) . We show there exists sequence {P ′ i } i ∈ N > 0 such that P ′ i ∈ C s ( η s ,i , P i ) for all i , and P ′ i →P ′ .

′ ′ ′ ′

Toward this end, for each ( η s ,i , P i ) , if h ( η s ,i , P i , P ) ≤ 1 , we set P i = P so that P i ∈ C s ( η s ,i , P i ) . If c i := h ( η s ,i , P i , P ′ ) > 1 , we construct P ′ i = ⟨ P ′ s ,i ⟩ s ∈I as the following. With Definition 2, there exist { p i 1 , . . . , p i n c ( G ) } and { p ′ 1 , . . . , p ′ n c ( G ) } associated with the interventional distribution tuple P i and P ′ respectively. Let P ′ i be associated with

so that P ′ i ∈ M G . For each s ∈ I and v ∈ Ω( V ) , we have

̸

where we apply the log sum inequality to ( a ) . Thus, for any s ∈ I , we have

As a result, h ( η s ,i , P i , P ′ i ) ≤ 1 c i h ( η s ,i , P i , P ′ ) = 1 , which means P ′ i ∈ C s ( η s ,i , P i ) .

It remains to show P ′ i →P ′ by construction. If h ( η s , P , P ′ ) < 1 , in the sequence { ( η s ,i , P i ) } i ∈ N > 0 , there exist finite instances such that h ( η s ,i , P i , P ′ ) > 1 . Otherwise, h ( η s , P , P ′ ) = 1 indicates c i → 1 , resulting in (16) →{ p ′ 1 , . . . , p ′ n c ( G ) } . Therefore, P ′ i →P ′ and C s is lower hemicontinuous.

With correspondence C s being continuous, Lemma 13 gives a continuous property of ˜ µ s ( C s ( η s , P )) . Lemma 13. For any P ∈ M G and for any any s ∈ I , the UCB index ˜ µ s ( C s ( η s , P )) is a is continuous monotonically nonincreasing function of η s ≥ 0 .

Proof. Since C s is a compacted valued continuous correspondence, we apply Berge's maximum theorem in Lemma 11 to get that max P ′ ∈C s ( η s , P ) P ′ s ( v ) is continuous with η s . So ˜ µ s ( C s ( η s , P )) = ∑ v ∈ Ω( V ) v n max P ′ ∈C s ( η s , P ) P ′ s ( v ) is also continuous with η s . For any a > b > 0 , since C s ( a, P ) ⊆ C s ( b, P ) , ˜ µ s ( C s ( a, P )) ≤ ˜ µ s ( C s ( b, P )) . Therefore, ˜ µ s ( C s ( η s , P )) monotonically nonincreasing with respect to η s .

The SCM-AAM utilizes the empirical interventional distributions to compute UCB and mean reward in exploration steps. Lemma 14 indicates that only a small ratio of exploration κ is wasted if the gap between empirical distribution and the ground truth is as small as δ ( κ ) . This fact is the cornerstone of regret analysis since the ground truth is unavailable.

Lemma 14. Given a causal graph G = ⟨ V , E , B⟩ , for any interventional distribution tuple P ′ = ⟨ P ′ s ⟩ s ∈I ∈ M G , let µ s ( P ′ ) be the mean reward of each intervention s ∈ I according to P ′ , and define µ ∗ ( P ′ ) = max s ∈I µ s ( P ′ ) . Let P = ⟨ P s ⟩ s ∈I ∈ M G be the true interventional distribution tuple with a unique optimal intervention s ∗ ∈ I . For each s ∈ I -s ∗ , let

in which C s ( η s , P ) := { P ′ ∈ M G | η s D ( P s ∥ P ′ s ) + ϵη s / 2 ∑ x ∈I -s D ( P x ∥ P ′ x ) ≤ 1 } . For any κ > 0 , there exists δ ( κ ) such that for any ¯ P = ⟨ ¯ P s ⟩ s ∈I ∈ M G , if ∀ : s ∈ I : ∥ ∥ ¯ P s -P s ∥ ∥ 1 ≤ δ ( κ ) ,

Intervention s ∗ is also optimal according to ¯ P , i.e., µ s ∗ ( ¯ P ) ≥ µ s ( ¯ P ) , ∀ s ∈ I -s ∗ .

For any s ∈ I -s ∗ , ζ s ( ¯ P ) ≤ (1 + κ ) ζ s ( P ) .

Proof. For each s ∈ I , it can be seen that lim η s →∞ ˜ µ s ( C s ( η s , P )) = µ s ( P ) since P ′ in C s ( η s , P ) is required to be identical to P as η s →∞ . For each s ∈ I -s ∗ , if ˜ µ s ( C s (0 , P )) > µ ∗ ( P ) , since ˜ µ s ( C s ( η s , P )) is a continuous monotonically nonincreasing function of η s according to Lemma 13, there exists η s > 0 such that ˜ µ s ( C s ( η s , P )) = µ ∗ ( P ) . Setting η ∗ s = max { η s ≥ 0 | ˜ µ s ( C s ( η s , P )) = µ ∗ ( P ) } , we have η ∗ s > 0 and ζ s ( P ) = η ∗ s . For any κ > 0 , ˜ µ s ( C s ( √ 1 + κη ∗ s , P )) < µ ∗ ( P ) due to the definition of η ∗ s . We define

From Lemma 12, C s is a continuous correspondence. Since D ( P s ∥ P ′ s ) + ϵ/ 2 ∑ x ∈I -s D ( P x ∥ P ′ x ) is a continuous function of P ′ = ⟨ P ′ s ⟩ s ∈I ∈ M G , it follows from Berge's maximum theorem in Lemma 11 that

is a continuous function of ¯ P = ⟨ ¯ P s ⟩ s ∈I . Thus, there exists δ ′′ s ( κ ) > 0 such that if ∀ s ∈ I : ∥ ∥ ¯ P s -P s ∥ ∥ 1 ≤ δ ′′ s ( κ ) ,

where the second inequality is due to the definition of C s ((1 + κ ) η s , P ) . The inequality means for any P ′ ∈ C s ((1 + κ ) η ∗ s , ¯ P ) ,

which means C s ((1 + κ ) η ∗ s , ¯ P ) ⊆ C s ( √ 1 + κη ∗ s , P ) . Therefore,

Let ˆ δ s ( κ ) = min { δ ′ s ( κ ) , δ ′′ s ( κ ) } , and let ˆ δ ( κ ) = min s ∈I -s ∗ ˆ δ s ( κ ) . Then we set δ ( κ ) = min { ∆ min , ˆ δ ( κ ) } , where ∆ min = min s ∈I -s ∗ µ ∗ ( P ) -µ s ( P ) . Since the reward is bounded by [0 , 1] , for any s ∈ I , ∣ ∣ µ s ( ¯ P ) -µ s ( P ) ∣ ∣ ≤ 1 2 ∥ ∥ ¯ P s -P s ∥ ∥ 1 . If ∀ s ∈ I : ∥ ∥ ¯ P s -P s ∥ ∥ 1 ≤ δ ( κ ) ,

which means intervention s ∗ is also optimal according to ¯ P . Besides, with (18) and (19), we have

Since ˜ µ s ( C s ( η s , ¯ P )) is continuous monotonically nonincreasing with η s and ˜ µ s ( C s ( ζ s ( ¯ P ) , ¯ P )) ≥ µ ∗ ( ¯ P ) ≥ ˜ µ s ( C s ((1 + κ ) η ∗ s , ¯ P )) , we have

which concludes the proof.

B.4 Main Proof of Theorem 2

The expected regret is defined as R SCM-AAM T = ∑ s ∈I -s ∗ E [ N T ( s )]∆ s , in which N T ( s ) is the total number of times s is selected by the SCM-AAM algorithm till T . So the proof is articulated around bounding E [ N T ( s )] for each s ∈ I -s ∗ .

Proof. At each time t , consider the event that the true interventional distribution tuple P = ⟨ P s ⟩ s ∈I is not contained by the confidence set C t = { P ′ ∈ M G ∣ ∣ ∣ ∑ s ∈I N t ( s ) D ( ¯ P s ,t ∥ P ′ s ) ≤ (1 + γ ) ln t } . Due to Corollary 8, there exists c 1 > 0 irrelevant with T

which will only result in finite regret bounded c 1 since reward is in [0 , 1] . So by only considering time steps with {P ∈ C t } to be true, we upper bound the number of times a suboptimal arm s ∈ I -s ∗ is selected in the exploration and exploitation phase separately.

Exploitation phase: When {P ∈ C t } is true, we have

̸

According to the design, SCM-AAM enters the exploitation phase at time t if ¯ µ S t ,t > ˜ µ s ( C t ) , ∀ s ∈ I \ { S t } . It follows from (21), if S t = s ∗ and P ∈ C t ,

Thus, we can bound the regret from exploitation steps as the following

Let { t s ,i } i ≥ 1 be the sequence of rounds such that s is selected by exploitation steps, namely S t s ,i = s . Considering the warmup step that selects each intervention once, at each t s ,i , we have N t s ,i ( s ) ≥ i . Since each t s ,i is a stopping time, by applying Lemma 9, we bound the expectation of (22) as

Exploration phase with inaccurate ¯ P t : We define event E t that ¯ P t is estimated with sufficient accuracy,

where δ ( κ ) > 0 is defined in Lemma 14. Let R = { t i } i ≥ 1 be the sequence of rounds that SCM-AAM enters the exploration phase, and each t i is a stopping time. Then we have

From Lemma 6, at t i , we have ∀ s ∈ I : N t ( s ) ≥ ϵi/ 2 . With Corollary 10, we apply a union bound

Greedy approximation exploration with accurate ¯ P t : For each s ∈ I , let N e t ( s ) be the total number of times s is selected in exploration phases till t . Within time steps such that both E t defined in (24) and {P ∈ C t } are true, let the last time s ∈ I -s ∗ being selected by exploration with greedy approximation be

is true. With Lemma 6, we have ∀ x ∈ I -s : N t s ( x ) ≥ ϵN e t s ( s ) / 2 , so that

Recall the definition of C t and C s ( η s , ¯ P ) :

With (26), we have C t s ⊆ C s ( N e t s ( s ) (1+ γ ) ln t s , ¯ P t s ) , and as a result,

By the algorithm design, we have ˜ µ s ( C t s ) ≥ ¯ µ s ∗ ,t , since otherwise SCM-AAM does not enter exploration phase. Therefore, it follows from (27) that

where the inequality holds since when E t s happens, Lemma 14 shows s ∗ is also optimal according to ¯ P t s . With Lemma 13, ˜ µ s ( C s ( η s , ¯ P t s )) is a continuous nonincreasing function of η s , then we have

where ζ s ( ¯ P t s ) defined in (17) is the maximum value of η s such that ˜ µ s ( C s ( η s , ¯ P t s )) ≥ µ ∗ ( ¯ P t s ) . Together with Lemma 14, we have

Forced Exploration: By slightly abusing the notation, at time t , we denote by { S t } as the greedy approximation exploration event and by { S t } as the force exploration event. Let the last time s ∈ I be selected by forced exploration be t s := max { t ∈ R | S t = s } . When SCM-AAM force exploration at t , we have N t ( S t ) ≤ ϵN e t , resulting in

We want to provide an upper bound N e T , which is

Also with (28), we have

Then, the expected number of forced explorations can be bound as the following, where C 1 and C 3 are defined in (20) and (25) respectively.

Summary: With reward V n ∈ [0 , 1] , we decomposed the expected regret as

where the first three terms present regret from exploitation, exploration with greedy approximation, and forced exploration respectively. So the proof can be concluded the proof by substituting (20), (25), (23), (28) and (31) into the above inequality.

C Experiment Details and Additional Experimental Results

In the main paper, we conducted experiments in three causal bandit instances. The causal graph corresponding to each instance is shown in Fig. 2. For p ∈ [0 , 1] , let B ( p ) denote a Bernoulli random variable with probability p to be 1 and 1 -p to be 0 . The detailed structural equations for each causal bandit instance are described below.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents two primary contributions for the causal bandit problem with unobserved confounders, general graphs, and discrete RVs.\n1) a lower bound on the regret any algorithm can obtain for this problem. \n2) an upper bound on the regret of a newly proposed algorithm that is able to generalize across interventions (unlike an algorithm such as vanilla UCB) using allocation matching.\n\nSTRENGTHS:\n- This is the first causal bandit algorithm I've seen that has a regret guaratnee for the cumulative regret setting and general graphs (besides [20], which focuses only on reducing the number of arms rather than using structure to generalize across arms). I think the community will find this very interesting. \n- Generally well written prose.\n- There is not much to add regarding the strengths: the paper is generally clearly presented and the result is a novel solution to a open problem that the community has shown a good amount of interest in. I have not gone through the supplementary material so I cannot comment on the correctness of the proofs.\n\nWEAKNESSES:\n- Experiments are synthetic (fairly common in these causal bandit papers in fairness) but also on extremely small graphs. It would be good to understand how performance varies with graph size and the size of the intervention set. \n- It would be nice to see a discussion in words of how the regret bound scales with various quantities- graph size, intervention set size. The bound as presented is not one of the easiest in terms of extracting this kind of qualitative information. \n- Some prose was clunky and I recall a few typos. The text should be run through Grammarly or a similar tool.\n\nQUESTIONS:\n- Line 293 onwards compares the regret guarantee to [20]. Is it possible to state exactly that the regret bound of your approach is upper-bounded by theirs? I couldn't directly infer this from the comparison you gave. \n- The algorithm seems to be anytime (doesn't need to know T) but would be great to have this clarified explicitly. \n- Also seems like you can have arbitrary constraints on the interventions and also that multi-target interventions can be used in your algorithm. Is this correct? \n- How computationally efficient is the algorithm? In the computation of the UCB (in algo 1 and remark 2) it looks like you sum over all possible configurations of the variable in the graph, which will explode as the number of nodes gets larger. The graphs in the experiments are also very simple/small so I'm wondering how scalable this method is. \n- Can you clarify what and how many additional hyperparameters the method requires compared to UCB? How do you set these hyperparameters in the experiments of the main paper? Look like you additionally have just $\\epsilon$ and $\\gamma$. \n- Standard UCB can be used without a forced exploration step- why/how does your algorithm break without it? I'm more familiar with regret guarantees that are independent of $\\Delta_s$ (so where regret is on order of sqrt(T)). What happens here as $\\Delta_s$ gets very small or very large?  The forced exploration likely explains why for small time horizons the proposed algorithm is not as good as the baselines on some graphs.\n\nLIMITATIONS:\n- In the related work the nonparametric setting studied here is presented as a benefit but it should also be mentioned that the fully discrete setting here can be limiting compared to works that study continuous observations (it is also not really clear that the main ideas described here can be easily extended to those settings). \n- The computational intensity of the algorithm, especially depending on graph size and intervention set size, should be discussed. \n- Line 338 \"it is shown that SCM-AAM has superior empirical performance compared to existing baseline algorithms\". I think this is potentially misleading since the baseline compared to is from one of the earliest casual bandit papers and likely not the strongest among published baselines on the graphs used. I will not insist on comparing against every or any other causal bandit algorithm published since then but I think the claim could be made more moderate, rather than implying that some kind of SOTA empirical performance has been shown.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper studies the structural causal bandits in which the causal graph contains latent confounders. The paper contains a lower-bound on attainable regret as well as an algorithm that achieves instance-dependent upper bound.

STRENGTHS:
The paper studies an established problem, building primarily on [20] and the idea of intervening on POMIS. The paper develops a number of theoretical results on attainable regret.

WEAKNESSES:
The paper is quite dense and could benefit from a more clear exposition on its theoretical contributions. For instance, this may include close-form or approximate values on the various factors listed in L228, L291.

To underscore the advantage of SCM-AAM and to establish novelty, it seems that one has to establish SCM-AAM is better than the algorithm developed in [20]. The comparison part L293-303 (particularly L298-L301) could benefit from greater clarity on the precise benefit of this paper's algorithm over that of [20].  It would be nice to see a particular factor, say in a particular causal bandit instance where it is clear the algorithm attains better regret. It seems like graphs in Figure 2 are promising candidates, as it is shown empirically that SCM-AAM does better.

QUESTIONS:
Please see above.

LIMITATIONS:
Seems fine.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method for studying structural causal bandits with unobserved confounders. They show that their method, which unlike the baseline (POMIS), includes two tuning parameters, has superior empirical performance on three synthetic experiments (three synthetic SCMs).\n\nSTRENGTHS:\nThe main review can be found in the Questions section.\n\nWEAKNESSES:\nThe main review can be found in the Questions section.\n\nQUESTIONS:\n## Abstract\n\n- Bandits do not have to be used online, offline usage if common as well. Please consider revising the first line.\n- An SCM does not model random variables as such; it specifies a distribution over the exogenous variables in the model (the noise variables) - but the endogenous variables are not random but are specified by the design and typically user experience.\n- Perhaps it would be better to say that the reward variable (everyone uses Y) returns the stochastic reward instead of the non-manipulable variables, of which there can be many but causal bandit only has one reward variable.\n- It is called the ‘exploration-exploitation’ trade-off, please fix this in your abstract.\n- I would urge the authors to considerably shorten their abstract. It is meant to short summary of your method and main results. I fear that at present length it is eating into valuable space in the paper, space which could be better spent on explaining your method. \n\n## Introduction\n\n- I think a bit more precision is in order in this section, you say on line 54, that “it [causal inference] has special value”. This is true, but perhaps it would be better to explicate on what that value actually constitutes and the utility it has compared alongside purely associative statistics (correlation-based statistics).\n- Lines 63-70 require _a lot_ of background knowledge to be read with any sort of utility. The POMIs, for example, requires an entire paper (Lee and Barenboim) to expose properly or you could just state the main result. But you haven’t instead you had merely mentioned POMIS but without telling the reader what it is order what it does. That is not very helpful particularly since the POMIS is such a powerful concept. Equally you have no adequately exposed the KL-UCB algorithm beyond stating it with its name. That does not do it justice and certainly not in the causal bandit context.\n- On line 69 you say: “A general linear model is adopted in [12], which is different from the non-parametric setup in this paper”  - but _how_ is it different? What is the non-parametric setup in this paper? What is a linear model in this context? You need to be much more precise in order for your readers to follow your line of reasoning.\n- On line 72: how can the arms be related to an arbitrary causal diagram? They are functions of the topology and that in itself is a function of the structural equation models specified in the SCM (under Pearl’s models of causation). The _domains_ of the arms may be arbitrary (though that would be rather odd). Hence, I am not quite what you mean by this sentence. Please do explain.\n\n## Problem Statement and Background \n\n- I am not sure you even need to state assumption 1, it is such a common assumption with causal bandits (indeed most structural bandits) that it is left unsaid and assumed given.\n- I think you mean to say on line 132: “means the expectation is computed with [respect to the] probability distribution P”.\n- Is there a specific reason you name your reward variable $V_n$?\n- Line 150: what are you actually referencing with [36]? Your reference is an entire book but what more precisely are you giving reference to?\n- With assumption two, are you suggesting that there are instances where the POMIS does not contain the optimal set or two or more sets in the POMIS have the same reward? You suggest in the line above the assumption that certain SCMs do not necessarily obey this assumption? Could you please explain further what you mean as this seems to contravene the results by Lee and Barenboim’s work on intervention sets.\n\n## Response Variables and Space of Interventional Distribution Tuples \n\n- Please explain how $V_2$ is a c-component, I do not follow; it is not connected to any bi-directed edges in figure 1(a)\n\n## SCM-based Causal Bandit Algorithm\n\n- SCM-AAM has two tuning parameters compared to the more canonical POMIS selection (merely a function of the topology) - it seems like you are making things more complex by introducing tuning parameters rather than just leveraging the topology of the DAG, as is the case with the MIS and POMIS. Can you please comment on this.\n- I really like how this section is exposed and told. Good job.\n\n## Finite time regret analysis\n\n- Please comment on settings where G does not contain any confounders (dashed edges). Is there any utility by using SCM-AAM then? Or do the results then fall back to those in reference [20]?\n\n## Experimental results\n\n- I think it would be more fair for you to use the Thompson sampling and KL-UCB algorithms instead of vanilla UCB without structural information, that does not sound like a reasonable choice in this setting. Or even the algorithms used by Lattimore and Lattimore in their 2016 paper.\n- Why do you choose those values for your hyperparameters gamma and epsilon? Have you conducted an ablation study for different variable settings?\n- The shape of the purple curve is not particularly surprising since it does not have access to causal information. As noted; this is not a fair comparison and you should leverage other, fairer methods, for comparison just to make it an even ‘fight’. There is very little to learn from the purple curve given that Lee and Barenboim have already shown that algorithms with causal information perform better than vanilla algorithms (Lattimore and Lattimore also commented on this idea).\n\nLIMITATIONS:\nThe main review can be found in the Questions section.\n\nI think the title is a bit misleading; plenty of studies have considered causal bandits with unobserved confounders (Lee and Bareinboim being a pair of authors who have published a number of works in this area). Why do you draw attention to this aspect of your method? Would be interested to hear your thoughts on this.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper considers online learning in a structural causal bandit whose stochastic environment is govened by causal relations represented by an SCM. The paper gives a logarithmic asymptotic regret lower bound characterized by an optimization problem, and inspires on that, provides an algorithm to untilize the causal structure and accelerate learning process. Numerical experiments illustrate that the proposed algorithm outperforms traditional online learning bandit algorithms.

STRENGTHS:
1. Writing. The paper has a clear organization of the relative background, concerning Structural Causal Model, Causal Bandit, and Interventional Distribution Tuples in Section 2-3. The presentation is good and easy to follow.
2. Originality. The paper studies online decision-making algorithms for causal bandits. The lower bound result extends from similar results in the structured bandit [1], and so as corresponding algorithm.
3. Quality. The writing is of high quality, and theoretical results and experiments are enough to support the algorithm. 

[1] R. Combes, S. Magureanu, and A. Proutiere. Minimal exploration in structured stochastic bandits. Advances in Neural Information Processing Systems, 30, 2017.

WEAKNESSES:
1. The whole paper seems to extend the methodology and algorithm in [1] to causal bandit settings. It is better if the author could highlight extra contributions of this paper.

[1] R. Combes, S. Magureanu, and A. Proutiere. Minimal exploration in structured stochastic bandits. Advances in Neural Information Processing Systems, 30, 2017.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23